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FOREWORD  J 


This  Planning,  Management  and  Evaluation   (PME)   Guide  has 
been  developed  by  the  National  Alliance  of  Business  as  part  of 
its  program.iD|  management  assistance  for  Private  Industry  Coun- 
cils.'  The  Guide  is  intended  to  be  a' tool  which  a  PIC  can  use  to 

•  improve  its  capability  to  plan,  manage,  # 
.and  evaluate  the  programs  which  it 

administers  •  4 

•  establish  locally  defined  business-' 

oriented  performance  standards  for 

Title  VII  programs. 

• 

The  Guid^  and  the  planning,  management  and  evaluation 
process  which  it  presents  ar?  being  field  tested  in  this  fiso&l* 
year, with  ten  PICs:  ^  ^ 

•  The  Boston  Private  Industry  Council,   Inc.  I 

•  The  Industry  Council  of  Gteater  Cincinnati,  Inc. 
J    •     The  Missouri  BsriaAce  of  State  Private*  Industry 

Council  s  i 

,  •     The  Private  Industry  Council  of  Atlanta,   Inc.  ^ 
The  Broward  Private  Industry  Council  (Fla.) 

•  The  Private  Industry  Council  of  the  iehigtt 

Valley  (Pa.) 

•  The  Buffalo>an£  Erie  Private  Industry  Council 

•  The  Private  Industry  Council,  Jobs  for  Fall  • 

River  (Mass.) 

•  -The  New  York  Balance  of  State  Private  Industry 

Council,  Inc.  . 

•  The  Delaware  Private  Industry  Council  r~ 

Based  upon  that  field  (test,   the  process  and  the  Guide  Will  be 
revised,  refined,  and  then  packaged  in  final  form  for  national 
dissemination,  fc 

The  Guide  can  also  be  useful  to  Prime  Sponsors  as  they 
administer  programs  under  other  titles  of  the  Comprehensive 
Emp logmen t% and  Training  Act   (CETA) .     It  has  been  directed  to 
PICs  and  PSIP  because  PICs  hav.e  been  assigned  a  unique  role  to 
involve  lousiness  in  public  employment  and  training  programs. 


\ 

In  so  doing  they  must 

•    chart  their  own  special  course  and  that  of 
business  in  the-  community 

*  •     involve  the  business  community  so  that  it 
can  take  an  active  part  in  local  programs, 
not  only  by  servLng  on  the  PIC  but,  also  by  pro- 
viding other  resources  to  complement  ttfose 
publicly  provided 

•r   embark  on  many  new  activities,  often  diffi- 
cult to  measure,  which  should,  be  undertaken 
V        with  care  to  assure  that  their  purpose  is 
l     •  clear  and  their  results  cao  be  assessed 

<  •     identify  ways  to  assist  all  publicly-funded 

,  '  employment  and' training  programs  to  provide 

opportunities  in  the  private  sector. 

To  assume  these  responsibilities,   the  PIC  needs  to  understand 
-thoroughly  what  the  Prime  Sponsor  and  others  bAse  accomplished 
and  are -trying  to  do,  and  what  business  needs  and  what  it  can  do-  to 
help.     The  PIC  must  then/be  able  to  determine  if  the  steps  it  took 
to  meet  these  needs  wereSruccessf ul .     With  such  responsibilities,. 
NAB  is  convinced  that  PICs  must  have  an  adequate  PME  process. 

Finally,  NAB  believes  tkat  the^ private  sector  can  be  parr 
* ticularly  helpful  to  PICs  and  Prime  Sponsors  in  .establishing 
PME  systems.   Therefore,  NAB  has  directed  this  initial  effort 
to  PSIP  where  . business  is  generally  most  involved,  but  hopes  that 
prime  sponsors  will  also  find  it  useful  for  other  programs. 

The  'Guide  is  being  distributed  now  in  working  draf^  form/ 
so  that  all  the  PICs     across  the  country  can  use  it  as  an  aid^ 
and  reference  to  facilitate  .their  planning  and-system  develop- 
ment  efforts  for  Fiscal  Year  1982  .  •  , 
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INTRODUCTION 

f 

The   following  are  principal  assumptions  underlying  the 
development  of  this  fcuide : 


PlCs  have  had  to  -devote  a  majority  of 
their  time  and  energy  to  getting' 
themselves  organized ,  establishing 
their  administrative  structure  and 
designing  andv  implementing  their  initial' 
programs.     As'  a  consequence,   few  PlCs 
have  had  the  time  *to  initiate  comprehen- 
sive'and  systematic  planning,  management, 
and  evaluation   (PME)  ^efforts  . 
•> 

The  planning,  management,  and  evaluation 
which  has  taken  place  is  that  required 
under  the  law.-    Plans  and  reports  have 
primarily,  been  external Ly-ori£nted  docu- 
ments prepared'  in*  order  X^o  secure  funds 
and  account  for  program  act/vities  and 
expenditures.- 

There  is  a  wealth  of  information  regarding < 
PME  systems  as  they. relate  to  employment* 
and  tra ining 'programs .     However,  much  of 
this  information  is  written* or  fdrmated 
in  such  a  manner  that  it  has  been  either 
inaccessible  to* the  "average"  reader,  or 
difficult  to  translate  into  specific  terms 
for  application  within  the  local  setting. 
This  has  seriously  restricted  the  utiliza- 
tion of  more  rigorous  or  sophisticated 
PME  approaches  for  ensuring  effective 
and  accountable  program  operations. 


•  Many  of  the'  individuals  who  have  PME  re- 
sponsibilities Within  a  felC  yill  have  had 
little  or  noyffcior  experience  in  imple- 
menting PME  systems. 

» 

Thi,s  Gu ide  addresses  all  of  the  foregoing  assumptions. 
It  is  written  as  a  primer.,    It  assumes  lj.ttle  pre-existing 
knowledg^  of-  PME  on  the  part  of  the' reader.     The  Guide  attempts 
to  detail^,   in  a  complete  and  clear  jnanner ,  all  the  basic 
steps  to  he  taken,  questions ^o^be  answfered,  and  information 
to>l?e  considered  in  implementing  a  planning,  . management  and 
evaluation  system  within  a  Private  Industry  Council. 
V 


The  Guide  does  not  spell  out  in  detail  or  espouse  one 
specific  PME  system.     Rather,   it  provides  a  logical  framework 
and  thinking  process  which  any  PIC  can  use  to  design  an  inter- 
nally-oriented (  PME  system  which  is  most  appropriate  and  effec- 
tive  given  its  own  situation  and  needs.    v  * 

'The  Guide  is  intended  for  PICs  regardless  of  size  or 
structure.     Each  should  apply  the  process  tp  the  extent  its 
resources  allow. 

i 
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WHO  SHOULD  USE  THIS  GUIDE 


This  Guide  is  intended  for  all.  PICs    regardless  \ 
of  th^ir  organizational^  nature  or  present  developmental 
status.     The  Guide  has  been  designed  so  that  it  can  be\ 
used  by  any  PIC  to~review>  enhance/  or  establish  its 
approach  to  planning,  management,  and  evaluation. 

The  Guide 'is  directed  primarily  at  those  staff  who 
yill  hav^  the  major  responsibilities  for  designing  and 
implementing  a  PME  process  within  a  P,IC.     However,  it 
should  also  prove  of  value  tq>  Council  members'  in  ^gaining 
an  understanding  pf  PME  as 'it  relates  to  the  PIC,  and  in  , 
making  decisions  regarding,  their  roles,  responsibilities, 
and  involvement  in-their  PIC's  PME  process.     Sections  I 
and  II  of  the  Qtiide  should  be  especially  useful  in  this 
regard. 

Finally,   hopefully,  the  Guide  will  be  of  benefit  to 
anyone  who  works  on  improving  a  PIC's  planning,  management 
and  evaluation  process. 
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HOW  TO  USB  THIS  GUIDE     .  .  .        •  .  , 

z  t 

*  ^      *     ■"  *  * 
•*  ^  '     *  .*  , 

'    '     -  \      '  ,  '  .  ,  | 

This  Guide  has  been  developed  so  that  it  can" be  u£ed 
independently,  ox  with  assistance,   to  implement  a  s^sti^matic 
approach  to  planning ,  management,  and  evaluation.'    The  Guide 
consist^ of  five  major  sections,  a  Glossary    and   an  Appendix. 

•  Section  I  6f  the  Guide ' discusses  PIC  PME  ^ 
in  general  and  provides*  a  self-assessment 

.  .  checklist  which  can  be  'used  ^to  evaluate 
the  present  status  'of  tbe  PIC's  PME 
#  ^      ^    <    approach        *-  ?;  ^  * 

•  Section  II  provides  a  general  orientation 
and  introduction  to  the  PM?  process 

•    •       v  •       •  '  " . 

•  Section  III  detail?*  the  specific  phases 
\          \*  and  steps  in  the  jprocess  and  furnishes 

information  and  ^techniques  which  can  be 
employed  in  app/ying  the^  steps  in  the 
process 

•  Section  IV  contains  various  aides  whrch 
can  be  used  In  establishing  the  PME  system 
within' a  PIC  « 

.    •     Section  V  has  beert  set  aside,  "to  house 

important  documents  delated  to  the  PIC's 
PME  system 

•  The  Glossary  of  Terms  contains  the  defini- 

"  tions  of  t,hase  coj^/epts  and  terms  which  are 

central  to  the  PME  process 

•  'The  Appendix  cifes^ additional  material  which 

•        can  be  ref.erred  to  in  order  to  facilitate  the    *  • 
**  development  and  refinement  of  the  PIC's  PME 

system.  ' 

If  your*  needs  are  to  establish  or  substantially  modify 
your  PIC's  present  PME  process,   it  is"  recommended  -that  you 
carefully  read  'Sections  I-\III  in  order     answering  all  questions 
and  reviewing  all  information.     This  shduld . provide  the  know-. 
ledg£  and  experience  *  base  necessary  to  employ  the'  aids  presented 
in  Section  IV,  or  suitable  alternatives,   in  the  process  of  ' 
designing  your  PIC  1  s  PME  approach, 

>  a 


IX 


It,  on  the  other  hand-,  you  are  satisfied  with  your 
present  process,  then  two  optional  uses  are • suggested : 

1.  Use  ^he  self-assessment  test  presented 

on  pp.  It8,   9  to  decide  which  areas  to  read 
in  the  Guide  and  wjiere  to  concentrate 
.your  improvement  efforts.  .  %  ^ 

2.  Read. those  §ections  of  the  'Guide  which 
are  of  intete-st.  to  you  noifc*.  and  refef 
to  the  Guide  .on  an  as-needed  basis,  when 

'  problems  occur  or  questions  arise^whilS 
implementing  your*  PIC 1 s  standard  PME 
process .  c 
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private  industry  council 


NING,  MANAGEMENT  AND  EVALUATION 


/ 


Th^  major  injtent  of  the  Private  Sector  Initiative 
:  ♦  ♦  * 

Program  (P6l£) ,  Titl^  VII  of  the  Comprehensive  Employment 

ancf gaining  Act^  (CETA)   is:  ^ 

#     to  increase  the  involvement  of  the 
business  community  in , employment  and 
training  activities 

* 

to  increase  private  sector,  emplgtyment 
arid  training  opportunities  for  CETA- 
eligible  , participants . 

Your  Private  Industry  Council   (Pl^f^has ' been  designated  as 
.the  "primary  vehicle,"  at  the  local  level,  working  in  con- 
junction  with  your  CETA  Prime  -Sponsor ,   for  achieving  that 
intqnt. '  .  • 

Specifically,   it  is  expected  that  your  P.IC  will  take 
a  leadership  role  in:  *  ~ 


'A 


shap 


directing,  and  coordinating  PSIP 


assessing  N:he  needs  of  local  industries 
and  ^empl6yers 


•  ' '  1 

iployment  and. 


developing unno vat iVe  employmenl 
training  activities  and  services 


e  activities  to  meet  the 
of  the  \Locai  brffciness 


design.i 
specif i 

community 

■&  — 

improving  existing  programs  and  methods 
for  the  delivery  of  employment  and  train- 
ing services  '  > 
& 

reviewing  critically  the  employment  and 
training  plans  of  the  CETA  prime  Sponsor 
and  others  to  ensure  that  they  are 
responsive  to,  business  and  community  needs 

bringinqJ  private^ resources  to  bear  on 
employment  3nd  training  is^^s  ♦ 

"1-1 
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•    \       .  • 

*       *  •    -reshaping,  as  necessary,  the  local  system 
or  network  for  the  delivery  of  employment  ^ 
and  trailing  services  s^y^ 

r     N  -  - 

•  forging  new  linkages  with  government 
agencies   (e,g'.  ,  vocational  education, 
housing,  public  welfare)   and  the  private 
sectors  to  contribute*  to  local  economic 
development • 

These  £xpfectc$tlpns  present  an 'imposing  challenge.'  Establish- 
ing"  an  effective  planning,  management  and  evaluation  (PME) 
,  (process  is  a  fundamental  step  that  your  PIC  can  take  tou enable 
you  to  .roee* .  the^t  challenge. 

Purposes  of  a   PME  Process  '  - 

r  ; 

A  PME  process  can  be  used  to  help  your  PIC  : 

•  develop  high  quality  and  effective  programs 

•  mAage*  the  performance  of  your  PSIP 

•  comply  with  federal  regulations - 

PME  for -.Program  Development 

One  of  the  overriding  reasons  for  the  existence  of  the 
PSTP  is  to  encourage  programmatic  experimentation  and^ to  pro- 
mote changes  within  the  existing  system  for  the  delivery  of 
employment  and  training  services.     In  recognition  of  this  fact, 
, 'your 'PIC  has  been  given  enormous  latitude  and  discretion  to 
forge  a  program  of' service  and  activities  whi-ch  is  best  suited 
.to  local  needs  and  which  has  a  high  chance. of  success. 

To  develop  such  a  program,  your  PIC  has  to  have  a  way 


to  : 


d  . 

•  identify  what  business  and  community  needs 
are  ^ 

•  pinpoint  existing  problem  areas  or  barriers 

•  decide  what  needs  to  be\done 

ascertain  which  existing  projects  work  and  s 
which  don't 

•  establish  a  comprehensive  approach  to  pro- 
gram development 

•  create  new  projects  to  mefet  presently  , 
unfilled  needs  » 
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*•     secure  feedback  on  those  programs  which 
are  initiated.*  %' 

A  well-designed  PME  process  can  serve  these  ends." 

♦  •  •*• 

PME  for  Performance  Management 

A  PME  process  also  ajovides  the  basis  for  directing  and 

controlling ' the  program  wnich  your  PIC  decides  to  implement. 

You  can  use  a  PME  process  "to  : 

•  set  overall  goals  and  operational  strategies 

•  establish,  program  performance  standards 

•  define  management  ai^d  monitoring  methods 

•  ^  determine  corrective  ^action  procedures 

•  ensure  accountability  in  program  delivery 

•  obtain  evalUat^n  data  which  can  be  used  %  " 
to  improve  program  performance.  r 

*  *• 

PME  for  ^Compliance  . 

Finally,  your  PIC  canftise  a  PME  process  to  ensure  satis- 
factory  compliance 'with  f ederal _ regulations .     The  CETA  regula- 
*    tions  require  that  a  Prime  Sponsor  submit  an  Annual  Plan-  which 
describes  all  of  the  jprograms  it  will  undertake  in  the  upcoming 
fiscal  year.     The  Titr^VII  subpart  of  this  plan  must  detail 
the  activities  that  will  be  sponsored  through^your  PIC. 

The  regulations  stipulate  that  the  subpart  must  consist 

of  a  Narrative  Description,  Statement  of  Concurrence,  Program 

•»     "  , 

Planning  Summary,  and  Budget  .Information  Summary.  The  contents 
of  these  items  are  described  briefly  below: 

Narrative  pescription  includes: 

•  goals  and* objectives  for  Title"  VII 

-  •  -program 

•  analysis  of  ^rffsi stance  needs  wi thin 

common It. 

•  target  grpo5>£  for  service  through 

4    Title  fxr*     m  > 

^#     activities  to  be -Undertaken  to  achieve 
objectives 

•  system  to  be  used,  to  review  and  assess 

the  success  of  the  activities 
results  and  benefits  to  be  obtained 

-  •  '  through  program   (quantitative  and 

*  qualitative  outcomes  expected), 

PIC  responsibilities  and  procedures. 
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Statement  of  Concurrence \     documents  the 
'concurrence  of  the  PIC  and  the  Prime  Sponsor 
with  the  content  of  the  Annual  Plan'  subpart. 

Program  Plan-  SummaiTy  (PPS)  :  presents  the  . 
goals,  objectives,  and  activities  planned  m 
for  the  program  year. 

Budget  Information  Summary  (BIS):     sets  out 
the  planned  budget  for  the  program  year. 

"Obviously,  being  able  to  prepare  a  plan  of  the  type 
required  by- regulation  necessitates  that  your  PIC  have  a 
wprkable  PME-  process  in  place. 


J 

CRITICAL  PME  DECISIONS  % 

There-  are  three  fundamental  ^ecisions  which  will  have 
a  substantial  and  significant  impact  on  the  shap^  of  your 
PME  process  and  the  rranner  in  which  your  PIC's  program  is 
directed  and  operated  generally.     These  decisions  are: 

•  What  perspective  or  approach  does  your 
PIC  want  to  fcgring  to  its  involvement 
with  the  employment  and  training  sys- 
tem? 

*  • 

•  What  relationship  will  your  PIC  have 
with  the  CETA  Prime  Sponsor? 

•  At  what  points  will  your  Council  be 
involved  in  your  PIC's, planning , 
management  and  evaluation  process?  4 

PIC  Perspective  and  Approach 

The  CETA  regulations  establish  only  broad  parameters 
and  minimal  expectations  regarding  a  PIC  in  terms  of  its 
existence ^and  operation.     Within  that  context,   it  is  up 
to  each  PIC  to  define  a  mission  and  orientation  which  is 
appropriate  to  local  circumstances  and  needs.     Therefore,  % 
the  principal  decision  which  your  PIC  must  make  is  how 
*it  can  best  be.  involve^  and  what  general  approach  i t > 
wants  to  take  to  working  within  the  local  employment  and 
training  system.     That  involvement  can  vary  along  a  con- 
tinuum from  bqing  purely  advisory  at  one  end  to  being  an 
operator  of  employment  and  training  programs  at  the  other. 

Factors  and  characteristics  which  should  be  taken  into 
account  in  determining  the  type  and  extent  of  your  PICfs 
involvement  include: 

•  limitations  imposed  by  the  amount  of 
available  resources  * 

*  > 

•  the  "perceived"  status  and  capabilities 

of  the  local  CETA  system 


•  local  economic  conditions  \ 

« 

'  •  present  strength  and  productiveness 

of  private  sector  ties  .  , 

•  local  political  conditions 

Regardless* of  the  specific  perspective  and  approach- 
which  your  PIC  jsej^scts,  a  basic  responsibility' of  the  PIC 
is  to  ensure ^hat  the  private  sector ^  interests  are  repre- 
sented  through  the  PIC,  and  th*t  the  PIC  supports  programs 
which  benefit  both  the _£ETA-eligible  participants  and  the* 
business-  community. 

*CETA-PIC  Relationships  ^ 

In  order  to  makimize  PIC  effectiveness,  *the  CETA 
regulations  require  linkages  and  relationships  between  the 
PIC  and  the  prime  Sponsor.     The  regulations  anticipate  a  r* 
"partnership"  between  These  two  bodies  in  'addressing  local* 
employment  and  training  problems.     However,  as  with  the  ( 
mission  and^  roles  of,  the  IMC.  itself ,   the  nature  of  those 
linkages  and  that  partnership  is  not  mandated  but  left  to 
local  discretion.     As  a  result,   angther  essential  decision 
for  your  PIC  is  to  'determine  what  operational  arrangements  1 
to/Hake  and  how  best^to  work  wit*  the  Prime  sponsor.* 

Structural  options  range  from  being  ap  integral  part 
of  the  prime  sponsorship  with  snaregi  staff  and  facilities 
£o  being  a  separate  entity  with  semi-autonomous  status. 
Your  PIC  should  'decide  which  arrangement  is  best/  given 
local  needs.     The  important  thing  to  ensure,  regardless  of 
the  specific  methods, chosen,   is  that  your  PIC  have  cooperative 
working  relationships* with  the  Prime  Sponsor. 

Thes£  relationships  are  especially  'important/ when  it 
comes  to  designing  and  implementing  an< effective  PME  process.* 

*The  ?rime  .Sponsor  repre^3fits  an  invaluable  source  of  exper- 
ience,  information,  and  assistance rwhich  your.PIC  can  draw 

# 

.upon  both  to  take  jbhose  steps  necessary  to  create  a  process  , 
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'which  is  utilitarian,  and  to  apply  that  process  in  such  a 
wanner  as  to  ^produce  the  results  desired  by  your  PIC. 

»  « 
.Council  Involvement  Points 

'  >*     The  last  major  decision  which,  your  PIC  needs  to  make 
is  to- determine  at  what  points  the  Council  will  be  involved" 
or  will  intervene  in  the  planning,  management  and  evaluation 
"•process*"   You  A^ed  to  determine  Y\ow  and  in  what  way  the 
Council  wants  to  influence  programming  and  operations 
and  then  to  establis-h  a  mechanism  and  process  for  ensuring 
•that  influence  on  a  continuous  basis .     To^do  this,  the 
Council  needs  to:  8  - 

•  formally  define  and  articulate  its 
role  as  planner,  manager,* and  eval- 

uator  *^« 

•  organize  and  develop. the  capability 
to  fulfill  that  role 

•  establish  action  steps   a/id  an  agenda 
for  ensuring  the  accomplishment  Of 
that   r'ole.  * 

As  with  thfc  decisions  on  PIC  persp$cti\#£  and  approach 
and  CRTA-PIC  gelations,   what  your  PIC  decides  to  db  an^ 
Ijow  to  do  it  shoitid  be  based  upon  local  qeedsY~VAs  a  rule, 
however your  Council  should  attempt  to  involve  itself  to 
the  maximum  feasible  extent  in  order  to  have  the  greatest 
influence  and  control  over  plan's,  the  design  of  programs, 
and  their  operation.     Key  points  of  involvement  or  inter- 
vent ion  include : 

^  •  defining  the  PIC's  basic  miss>^p  and 
functidns 


•  establishing  PIC • operational  methods 
and  procedures 

•  identifying  the  needs  of  the  local 
business  community  1 

•  identifying* and 'selecting  occupations 
,  fo*  training  programs  *  ' 


•  determining  whicji  ttrget  groups  tq  > 
Serve  . 

•  deciding  what  services  or  assistance 
to  provide  to  tjiose  target  groups  ^, 

•  determining  what  other  services  or 
special  projects  can  be  initiated  to 
generate  employment  opportunities 
within  the  locral  community      '  ^ 

\  t 
*  1' 

»  establishing  priorities  for  services  and 
/activities  to  be  provide'd  through  the  PIC 

'    •  setting  strategic  and  operational 
./goals  for  the  P'tC 

•  assessing  and  selecting  service 
deliverers    §  *  , 

•  approving  thS  plans  for  operating, 
managing,  'and  evaluating  the  PIC,j^  Program 

•  monitoring  and  reviewing  program  and 
contractor^per formance  at  regular 
intervals  to  assess  ef f ectivenass 

•  planning  and  approving  modifications 
to  the  basic  program  approach  as 
necessary  - 

\  evaluating  program  performance  at  the  4 
end  of  the  operational'  year  to  make, 
replanning  decisions. 


YOUR'  PIC'S  PME  PROCESS     .  '     .  " 

As  the  foregoing  discussion  attests,  your  PIC-' /  PME 
process  is  central  to  program  effectiveness  and  success, 
3*nd  ultimately  to  the  achievement  jt  the  intent  of  the 
Private  Sector  Initiative ^Program.     Therefore,  at  this  point, 
prior  to  reading  the  next  section  of  this  Guide  whigh  describes 
a  comprehensive  and  systematic  approachrto  JJME,  it  might  be 
worthwhile  to  assess  your  PIC's  present  PME  process.     Use  the 


scale  below  to  indicate  your  rating  of  your  PIC's  overall 


process  for  PME . 


6  5  4  3  /  2     -  1 

  £  -  .  ■  r  I, 

Excel-  Very  Good  Fair  Not     \     >  Poor 

lent  Good  ^  Very 

Good 


Reflecting  upon  your  rating,  list  below  those  areas 
in  "which  you  think  your  present  process  is  most  in 
need  of  improvement  and  describe  the  nature  of  the 
improvement (s)   that  you  feel  are  needed. 

•  Planning  -  method  for  determining  local 
^employment  and  training  needs  and  for 
deciding  what  services  and  projects  I 
to  initiate  in  response  to  those  needs 

•  Management-  -  method  for  monitoring  arid 
controlling  program  performance  and 
taking  corrective  actions  as-  necessary 
to  ensure  program  success* 

•  Evaluation  -  method  for  reviewing  and 
assessing  progr3m  and  contractor  effec- 
tiveness at  conclusion  of  performance^ 
period  to  make  informed  replanning 

s^ecisions  ^ 


Area ( s)  Improvement (s)  Needed 


\ 


■ , 

Use  your  rating  and  analysis  to  fo£us  your  reading  of 
Section  II  which  f ol lows .  * 


II.     THE  PLANNING,  hlANAGEMENT  AND  EVALUATION  PROCESS 

1 

The  Planning,  Management  and  Evaluation  «(PME)  process 
is  a  .systematic  method  which  your  Private  Industry  Council 
(PIC)   can  employ  to  establish  and  implement  its  Private  Sec- 
tor Initiative  Program.     The  process  is  Resigned  tQ  help^you 

ask  those  questions  and  take  those  steps  most  essential  for 
r 

effective  PIC  decision-making  and  program  operations. 


A  Decision-Making  Tool 

j         Your  PIC  can  use/the  PME  Frocess  in  deciding: 

•  What  local  Aeeds  are  -  the  nature  a-nd  dimension 
_ of  employment  and  training  needs  within  your 

community.  ^ 

•  What  your  PIC  wants  to  accomplish  -  its  purpose, 
roles,  and  goals. 

m"  What  to  do  -  the  specific  objectives  to  be 
achieved  and  projects  and^ctivities  to  be 
sponsored  through  your  PIC. 

•  How  to  do  it  -  the  methods  to  be. employed  in 
implementing  each  project  and  your  PIC's  total 
program. 

• 

•  How  to  support  it  -  the  relative  amount  of 
resources   (time,  people,  mc?ney)   to  be  dedicated 

•  to  each  area  of  activity. 


b 


How  to  control  it  -  your  system  for  monitoring 
progress  and  performance  and  taking  corrective 
action  as  necessary. 

How  to  assess  it  -  your  mechanism  for  *  evaluating 
results  knd  program  accomplishments. 

How  to  manage  it.  -  the  procedures  for  feeding 
information  back  'into  the  PIC  for  replannihg 
purposes . b 
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A  Procedural  Guide  , 

The  PME  process  is  structured  to  allow  your  PIC  to  make 
the  following  decisions  in  a  logical  order.  The  majr)r  stages 
in  the  PME  process  are:  t 

Ay    Assess  "Local  Needs 

B.  Define  PIC  Strategy 

C.  .Establish  PIC  Program  Scope'  1 

D.  Develop  Program  Operations  Plan 

E.  Develop  Management  Plan 

F.  Deve,loj5  Evaluation  Plan 

G.  Monitor  Program 

H.  Evaluate .Results 


\ 

v  '  ■  •  ( 

A  Management  Process 

The  flow  of  the  PME  process  is  shown  in~the  chart  on 
the  following  page. 

•   The  Remainder  of  this  section  discusses  what  your  PIC 
does  in  ,each  stage  of.  the  PME  process   (the?  steps  to  be  taken 
and  questions  to  be  answered).     Section  III  details  how  to 
apply  the  PME  process  within  your  PIC. 
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A/    ASSESS  LOCAIL  NEEDS 
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•Purposei 
Steps; 


To  deteiynine  the^current"  employment  and  training 
needs  within  youf  community.  ^< 


Analyze  Local  Labor  Market  Conditions 
*ar)d  Business  Npeds  ^^^v 

Analyze  Local  Population  Groups  'and  - 
Service  Needs  ^ 

Review  Programs  and  Plains  of  Other  ' 
Agencies/Service  Deliverers/Businesses 

i 


tcomes: 


Accurate  and  comprehensive  assessment  of  local 

-  economic  'ic^nds 

.  -  employment  needs 

-  business  needs 

-  training  needs  *\ 

-  problem  areas 

-  employment  and  training  programs 

and  plans 


4 


The  information  obtained  through  yCur  local 
needs  assessment  is  used  as  primary  refer- 
ence dataiin  designing  your  PIC's  program 
so  that  it  is  realistic,  addresses  those 
lpcal  nfeeds  which  are  most  pressing,  and 
complements  the  ongoing  or  planned  activities 
of  others.  " 
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1.     Analyze  Local  Labor  Market  Conditions 
//^^ffnd  Business  'Needs  ~— 

m 

The  overriding  reasons  for  doing  labor  market  analysis 
*are  to   (1)   identify  potential  job  opportunities  within  the 

*  <w      ^       ,  m  c 

local  community  which  are  appropxiate  for  CETA  participants; 
and.  (2)  pinpoint  the  specific  needs, of  local  businesses  and 
industries  sb  that  your  PIC  can  develop  a  plan  based  upon 
these  factors.     '  t 

As  a  representative  of  the  business  community,  your  PIC 
is  uniquely  equipped  to  provide  arid/or  review  employer-defined 
labor  market  data  for  use  by  the  PIC*  and  others  tn  decision- 
making.    Data  gained  through  the  labor  market  analysis  can  be 
used  for  a  variety  of  purposes,  including: 

•  selecting  target  jobs  for  training  programs 

*  4  •  designing  customized  traioirtg programs  to 

^  +     satisfy  employer  defined  criteria 

•  identifying  obstacles  and  barriers  which  ' 
inhibit  the  effectiveness  of  employment 

f    \  and  training  programs  so  that  the^e  impedi- 

ments can  be  eliminated 
~*  ✓ 

•  determining  services   (e.g.,  employiifent 
development,  world  of  work  orientation, 
recruitment)  which  can  be  provided  through  > 
the  PIC  to^meet  local  employer  needs. 

*  The;  importance  of  J.abor  market  analysis  cannot  be  over 

estimated.     Although  you  will  hear  that  businesses  sometimes 
do  not  share  informati.on  on  their  labor  market  expectations  or 
do  not  dp  adequate  forecasting,   this  should  not  deter  you  from 
making"  every  ef*fort  to  collect  complete  information.     As  your-' 

.programs  increasingly  meet  business  needs ,  you  may  well  find 
that  busines^s  will  make  greater  efforts  to  provide  you  more 
complete  information. 

You  can  take  one  of  two  approaches  to  labor  market 
analysis.     You  can  rely  solely  on  existing  data  sources  (i.e., 
that  furnished,  by  the  CETA  Prime  Sponsor,   the  Labor  Market 
Information  Division  of  your  9tate  Employment  Security  Agency 
(SESA) ,  other  governmental  agencies,  and  universities).  Or, 
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you  can  complement  the  available  data  with  a  local  employer 
survey  of  your  own.  n  % 

Existing. data' sources  will  probably  be  sufficient  to 
help  you  determine  general  trends  and  answer  many  basic  ques- 
tions regarding  Ideal r 

•  labor  force  participation 

•  growth  potential  by  industry  and  occupation 

•  ' salary  ranglfcand  quality^ of  occupations 

*  •  skill  requirements  necessary  f&r  job  success 
-  ii>r various  occupations. 

Specially-developed  local  employer  surveys  enable  you 
to  determine  a  number  of-additipnal  aspects  of  the  local1  labor 
market,  including  the:  , 

•  nature  and  structure  of  the  internal  labor 
market '  Cjob  composition  and  entry  level 
occupations)  of  local  firms 

•  number. and  occupational  characteristics . of 
existing  job  vacancies  in  local  labor  markets 

•  hairing  practices  ^nd  policies  Of  local  firms 

r 

•  hourly  and/or  weekly  wages  'and  annual  earnings 
of  Workers  in  specific  occupations 

•  training  and  promotional  policies  of  local 
firms 

•  employer  attitudes  toward  existing  employment 
and  training  programs  funded  by  the  local  CETA 
Prime  Sponsor  and  other  local  employment  and 

^training  agencies. 

*  .  . 

Your  choice  of  how  to  do  labor  market  analysis  will  be 

determined  by:  > 

•  the  adequacy  and  accuracy  of*  the  data  avail- 
9  able  Xq  you 

•  -the  purpose  you  want  to  achieve  vwith  your 
analysis 

•  the  decisions  you  want  to  be  able, to  make.  ,y 

•   However,  since  local  employer  surveys'  are  easilyj dated ,  ( 
expensive,  time-consuming,  and  require  considerable  knowledge 
and  expertise  for  their  proper  implementation,  you  should  take 
the  following  steps 'before  initiating  one: 

v-  .  -n-e 


[  -       ^         Define  the  rinf ormatioji  need.     What  data  are     ,  / 
needed?    How  tfill  they  be  used  in  the  planning 
process?    Will  additional  data  help  you  to 
make  better  decisions? 

•  Utilize  existing  resources.     See  if  you^can 
get  the  additional  data  you  need  from  tradi- 
tional sources   (e.g.,  the  Prime  Sponsor,  the 
State  Employijtent  Security  Agency   (SESA)  ,  and 
State  Occupational*  Information  Coordinating 
Committee   (SOICC) ✓ 

v        •  Assess  the  survey.     Make  a  careful  assessment 

of  the  costs  of  the  survey,  correct  survey 
procedures*   reliability  of  data  to  be  gathered, 
+         and  possible  alternatives  to  the  survey/ 

•  Make  the  decision^    Reevaluate  your^Ktal  - 
information  requirements  to  see  if  they  can 
be  reduced,  whether  alternative  data  can 
be  used,  and  whether  certain  data  needs  can 
be  dropped  without  adverse  impact  on  the 

•      .  planning  prQcess.  % 

•  Coordinate  with  your  Prime  Sponsor,  SESA,  and 
SOICC.     Once  you  decide  to  do  a- formal  employer 

v  survey,  contact  your  SESA  and '.SOICC  to  take 
advantage  of  any  related  activity  and  to  secure 
.their  assistance  in  your  project. 

■% 

,  In  conclusion,  labor  market  analysis  is  the  cornerstone 
for  an  effective  PIC  program.     The  more  you  can  do  to  ensure 
that  your  analyst^  is  accurate  and  meets  yout  decision-making  * 
rie'eds,  the  more  probable  it  is  that  you  4will  design  a  program  capa- 
ble of  succes's.     In  addition,  by  sharing  the  results  of  ^your 
data  gathering  and  analysis  efforts  with  the  Prime  Spdhsor  - 
you  can  help  to  improve  the  Sponsor's  capabilities  in  this 
area. 


2  .     Analyze,  Local  Population 
Groups  and  Service  Needs  * 

The^  purpose  of  this  step  is  to  identify  those  population 
groups  within  your  community  who  are  in  need  of  employment- and 
training  services  and  to* document  the  relative  extent  of  their 


ERJC 


need.     Just  as  important,  th&  step* involves  pinpointing,  as  pre- 
cisely as  possible,  the  exact  impediments  arid  obstacles  to 
job  success  and  unsubsidized  employment  among  these  groups.. 
You  should  use  data  contained  in  the  Master  Plan  portion  of 
j the  Prime  Sponsor's  Comprehensive  Employment  and  Training 
Plan  to -secure  the  information  for  this  analysis. 

Fundamental  questions  thatLyou  want  to  be  able  to  answer 
in  this  step  ^are:  *  ~ 

•  What  is  the  general  level  of  employment 
and  training  need  within  the  local  com- 
munity? *  \ 

»  What  are  the  needs  by  population  group- 
(sex,   age,   race/ethnic  group;  sub-groups 
Such- as  Vietnam  Vets,  public  assistance 
recipients,  alcoholics)?  v  , 

•  Are  some  groups  mtre  in  need  than  others , 

•  What  apparently  accounts  for  the  needs 

within  each  group?  %  *  < 

The  answers  to  these  questions  will  later  help  your  PIC 
determine  who  it  want^to  serve   (what  categories  of  indivi- 
duals^ groups),  and  what  types  of  services  it  wants  to  provide! 
■  \ 


\ 

3.     Review  Programs  and  Plans  of  Other 

Agencies/Service  Deliverers/Businesses 

*     In  this  step,  your  PIC  identifies  all  those  employment 
and^ training  service  deliverers  within  the  local  community  and 
then  reviews  their  past  performance  and  future  plans  for  ser- 
vice.    The  purpose  of  this  step  is  to  pinpoint: 

•  gaps  in  the  local  service  delivery  ^system  • 

•  possible  areas  for  collaboration  with  . 
other  service  deliverers 

v  *  areas  in  which;  ybur  PIC  may  choose  to 

initiate  new  services  or  projects. 
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Programs  and  plans  which  you  should  review,  at  a  minimum, 
include  those  of: 

•  the  CETA-  Prime  Sponsor  -  v  * 
,  ,  •    Employment  ^Service 

•  educational/vocational  courses  run  by  y 
local  high  schools,  community  colleges, 

or  proprietary  institutions 

^  •    econQmic  development  and  other  employ- 
ment and  training  initiatives  sponsored 
through  the  f edera 1  government ;  by  the  " 
•1  local  government;  or  by  private  Companies 

-   or  corporations 

•  employment  and* training  efforts  pnder- 
taken  by  private  sector  firms  and  busi- 
ness organizations.  t 

In  reviewing  the  plans  of  others,  you  should  be  looking 

fQr : 

•  timely  data  on  target  groups  and  Local 
target  group  needs 

•  projects  which  have  a  high  success  rate 


/ 


potential  service  deliverers  who  provide 
services  efficient ly  . and  effectively, 


4 


^  Based  upon  your  PICfs  review  of  others1  plans ^you  can 

then  structure  your-  PIC fs  program  to  complement  other  efforts 
being  undertaken  at  the  local  level.     Possible  courses  of 
action  include:    •/  / 

I     .  :' 
m{  directing  PIC  programs  toward  target 

groups  and  potential  job  opportunities 

not  'being  addressee^  through  other  pro- 

*.  grams 

%  •     "piggy  backing"  by  ;using  PIC  funds  to 

-support 'programs  which  have  a  record 
of  demonstrated  siiccess 

•    cooperating  or  linking  up  with  other 
agencies  to  "joint  venture"  a  project 


X 
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selecting  successful  basic-  program  , 
designs  and  modifying  them  as  appro- 
priate for  implementation  through  the 
PIC  ^ 

identifying  employment  and  training 
service  areas  in  which  th6re  are  gaps 
and  developing  special  programs  to 
fill  those  gaps' 

cooperating  with  businesses  to  use 
facilities  and/or  staff  to  provide 
training.  t 


B.  '  DEFINE  PIC  STRATEGY 


4  * 


r — * 


C.  Emm* 

Prtfram 


O^DmWp 
Opwttna 


F.  Otwltp 
PUi 


v 

C.  Oivtltp 

C.  MtMW 

MlMftfftMt 
Nil 
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Purple:     To  develop,  using  £he  data  secured  in  the  needs 

assessment  and  an  analysis  of  zne  PIC's  programmatic/ 
organizational  and  administrative  performance  over 
the  preceding  year,  a  strategic  plan  which  .includes 
goals  for*  the  upcoming  year  and  those /goals  to  be 
accomplished  in  the  longer  term.  A 


e 


Steps :        ,1*.     Assess  PIC  Program  Performance 

2.  Assess  PIC  Organizational  and  Operational 
Performance 

3.  Establish  Initial  Needs  Priorities 

a.     Define  Service  Needs   (training  and 

development  and  employment  generating-) 

t  b.     Select  Target  Groups 

c.     Determine  Occupational  Training  Priorities 

4.  Redefine  PIC  Mission  and  Fui>ction 

« 

5.  Refine  Basic  Organizational  and  Operational 
Framework  *  ' 

6.  Formulate  Goals  ^and  Strategic  Plan  % 


ERLC 
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tcome:     A  written  statement  whifeh  clearly  articulates 
ypur  PIC's 

-  purpose 

-  role  definition 

-  functions  L 

-  organizational  and  administrative 

structure  and  operational  procedures  * 

-  annual  goals  and  their  priorities 

-  long-term  goals 

-  resource  allocation  -  . 


Use:  The  strategic  plan  establishes  "the  general 

program  direction  and  emphases  for  ypur-.  PIC 
and  furnishes  the  overall  parameters  used 
to  oversee  and  operate^your  program. 


J 
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1 .     Assess  PIC  Program  Performance 

This  step  is  taken  to  determine  how  wel£  your  PIC  per- 
formed programmatically  (i.e.,  in  terms  of  project  or  service 
activities) . 

o    In  reviewing  program  performance",  you  should  look  at 
your  PIC  at  three,  levels — overall,  by  component,  and  by 
service  deliverer  of  contractor — and  answer  four  baSic  ques- 
tions: * 

•  How  well  did  your  PIC  perform  in  th,e 
aggregate  -  'quantitatively   (i.e.,  in. 
terms  of  the  numbers  related  to  parti-'  ' 
cipants,  costs,  resource  utilization)? 

•  How  well  was  ydM^tot^l  program  implemented 
qualitatively  (^re.,  according  to  standards 
set  out  in  your  pdan) ? 

•  » 

•  How  well  did- each  component   (e.g.,  class- 
room training,  OJT,  marketing,  labor 
market  analysis,  economic  developments 
perform? 

„   •  •    What  was  the  relative  effectiveness  or 

performance  of  the  service  deliverers 

or  contracters  that  you  used? 
* 

At  a  minimum,  you  will  want  to  answer  these ^qufcstions 
by  comparing  the  actual  perf ormarrce  of  your  progranj  against 
that  which  was  planned.     Much  of  the  aggregated  quanticative 
data  for  performance  vs.  plan  review  can  be  retrieved  easily 
from  three  repor ts--Program  Status  Summary,  Summary  of  Parti- 
cipant Characteristics,  and  Financial  Status  Summary-7which 
the  Prime  Sponsor,  must  prepare  quarterly  and  annually  for  .the 
Department  of  Labor.     Data  for  the  more  qualitative  assessn\ent 
and  the  review  of  components*  and  service  deliverers  or  con- 
tractors should  be  secured  from  your  PIC's  own  monitoring #and 
reporting  system. 

In  addition,  as  time  and  funds  permit,  and  your  PIC's 

i 

decision-making  needs  dictate,   you  might  want  to  make  a 
more  in-depth  assessment  of  your  program's  effect  |by  looking 


r 
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at  areas  such  as»  economic  impact,  institutional  impact,  atti- 
tudinal  impact,  patterns  of  service,  and  cost-benefits.  Some 
questions  your  PIC  might  ask  in  these  areas^ioilow. 

Economio  Impacts  on  Participants 

iWhat  do  various  measures  of  placements  and 
-  terminations  show? 

•  What  do  varioujs  measures  of  w^.ge  gains  show? 

•  What  can  be  said  "about  the  quality  of  jobs 
developed? 

•  What  can  be  said  about  the  retention  of 
participants  in  the  jobs  they  obtain? 

♦  •  What  can  be  said  about  advancement  of  parti- 
cipants from  their  first  job  to  better  jobs? 

« 

Institutional  Impact  of  Title  VII 

•  Has  the  PIC  stimulated  new  business  partici- 
pation in  local  employment  and  training  *  efforts?  ✓ 

•  Has  the  PIC  stimulated  new  responsiveness  on  the  * 
part  of  "traditional"  employment  and  training 
deliverers  tc  the  needs  of  business? 

•  Is  there  a  "comprehensive  planning  approach 
.  between  differant  prime  sponsorship  areas  and 

other  PICs?  T—/ 
' •  Has  the  PIC  forged  new  linkages  with  local* 
public  or  private  sector  economic  development 
agencies  which  have  helped  to  retain  or^expand 
local  businesses,   thus  treating  jobs  for  CETA 
eligible  participants? 

v 

Attitudinal*  Impacts  of  Title  VII  ' 

•  How' has  the  PIC  program  affected  the  attitudes 
of  employers  toward  the  -program,   toward  ^employ- 
ment  and  training  programs  in  general,  and 

*  toward  eligible  clients  as  potential  employees? 

«  What  impact  has  the  PIC  program  had  on  the  atti- 
tudes and  motivations  of  participants? 

r 

Patterns  of  Service 

•  What  are  thd  characteristics  of  the  partici- 
pants being  shaved?     How  do  they  compare  to' 
regular^  CETA  clients,  goals,  and  some  state- 
ments of  "universe  of  need"? 

•  What  are  the  characteristics  of  the  businesses 
being  served?    Whfere  do  they  stand  in  the 
overall  economic  structure  of  the  area? 

Cost 

•  What  treasonable  cost-benefit  calculations 
can  be  made  about  some  of  the  above  impacts? 


Regardless  of  the  areas  you  look  at  .and  the  questions 
you  choose  to  ask,  your  programmatic  assessment  should  enable 
you  to  draw  conclusions  which  can  be  used  in  making  determina- 
tions such  as:  ^ 

*  •    Were  your  initial  planning  assumptions  v^alid? 

•  Were  your  initial  goals  and  objectives  w 
'realistic?  "  ' 

,  •    Were  your  program  and  projects  well  designed? 

•  Was  your  overall  programmatic  thrust,  or  mix 
of  services  and  projects  sound?>  . 

•  Had  you  anticipated  the  barrios  which  you 
encountered? 

•  Are  there  aay  components  to  your  program  * 
which  should  be  dropped?,    Given  increased 
support? 

•  Are  there  any  service  deliverers  who  are  , 
ineffective?     Highly  effective? 

Obviously,  the  exactness  of  your  assessment • of  "program  ' 
performance  will  be  highly  dependent  upon  the  quantity  and_ 
quality  of  the  data  which  you  have  available  or  are  able  to 
secure.     The"  important  point  at  this  time,  in  the  evolution 
of  your  PIC,  is  that  you  conduct  some  type  of  formal  perfor- 
mance assessment  of  programmatic  outcomes  to  obtain  a  basis 
for  making  replanrting  decisions.     This  assessment  will  enable' 
you  to  identify  barriers  to  effective- programming.     The  iden-  * 
tific^tion  of  these  barriers  will  enable  you  to  plan  for  their 
elimination.  ' 


2.     Assess  PIC  Organizational 
And  Operational  Perfprmance 

At  the  same  time  that  you're  assessing  your  PIC's 
program  performance,  you  should  determine  how  well  your 
PIC  performed  organizationally  and  operationally.  The 
purpose  of  this  inquiry  is  to  examine  the  Piq's  basic 
arrangements  and  methods  of  functioning  to  determine  their 
utility  for  future  -program  operatrans. 
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Areas^hat  you  should  look  at  in  this  step  include: 
PIC  Organization,  PIC  Operations,  PIC ^Procedures  and  Methods. 
Possible  questions  that  you  might  ask  yourself  in  6$ch  of 
these  areas  follow.  *  r  ■ 

PIC  Organization  '  % 
^      Ar£  PIC  goals  .clear understood,  aftd  shared  > 

-   by  members?         "  x  / 

■         •                                ,  ■ 

Ar6  the*  PIC  members  clear  on  the  role  they 

want  the  Council  to  pl.ay  with  reference  to 
the  Private  -Sector  Initiative  Program?" 

•  Aire  the  divisions  of  responsibilities  'among     *  s 
ancj  betweeh  PI£  staftf  and. members  clear?  *        ^  J 

•  Does  the  PIcTtrava-^a^complete  statement,  of 
mission  and  function?        *  -  ' 

•  Is  th^Cguncii  stisUctured  Jnost  effectively 
and  efficiently  to  facilitate  both  decision-  .  m 
making  and  program  oversight?    4      ,  *• 


~      PIC  Operations  -  *  ,  •  '  „ 

*m      Is  the  PIC  well-staffed?    Are  the ''PIC  and  the 
staff  assigned  to-  or  hired  by  it  working  pro- 
,  \       ductively  together?  ' 

•      Is  there  an  active  core  of  the  PIC  membership* 
-    lafge  enough  to  sustain  continuous  decision- 
making and  program  support  activity?  Can 
vdfcanpies  be  filled  in  a  timely"  fashion,  and 
are'  they  filled  using  criteria  that-  help 
*t  sustain  and  build  this  core  of  active  members? 

%     Are  PIC  relations  with  the  Prime  Sponsor  gobd 
enough  to  allow  the  PIC  to  have  any  leverage 
^  on.  the  whole' <:ETA  system?  ' 

Is  the  PIC  visible-  and  positively  Received, 
in  the  local  business  community*? 


PIC  Procedures  and  Methods 

»     Does  the  PIC  have ' administrative  procedures  / 

and  operating  guidelines  in  place?    Are  they 

useful  and  cdrrent?  '  '  /' 

♦ 

•  Is  thfe  planning  process  fpr  Title  VII  orderly  -' 
and  does  it  allow  enough  time*  to  make  reasoned 
decisipns?  *  ,  f 
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•  -Is  the  planning  process  structured  to  include 
individuals  lid  institutions  whose  support  for 

^       the  final  product  is  important? 

■  Does  th^  PIC  have  a  sysNh^matic  method  for 
monitoring— program  performance  and  taking' 
corrective  action  as  necessary? 

•t   a   (  Does  .th'e  PlC  HaVe  a  systematic  plan  for 

^evaluating  The  effectiveness  of  each  acti- 
vity and  its  tdtal , program? 

•  Is  the  evaluation  data  which  is  collected 
sufficient  in  quantity  aijci  quality  for  PIC  v 
decision-making? 

Again,  the  bas%p  purpose  f or»  £his  analysis,  as  with  the 
analysis  of  program • performance,"  is  to  identify  barriers  to 
effective  performan9e.  ' 

' Replanning  -questions  that  you  will  want 'to  be  able  to 
answer,  based  on  your  assessment  pf  your  PIC's  organisational 
and  operational  performance,  include 


Is  there  q0r\eed  to  reorganize  or  tar  re- 
structure the  Council-? 

*  *  ~ .  » 

Is  there  a  need  to  clarify  .staff  and/or 
member  roles  and  responsibilities?  -  P 

s  there  a  need* to  improve  staff  or  Coun- 
1  performance  either  in-,  teriys  of  work 
^         output,  relations  with  one  another,  or  ^ 

external  relations?  ,  V\  ,  f- 

•  Is  there  a  need  to  develop  new  Or  more 
detailed  administrative  procedures  and 

•  '       operating  .guidelines? 

•  Is  there  a  need  to  upgrade  the  PIC  planning, 
management  and "^valuation  process? 

»  •    Your  answers  to  these  questions  will  begin  the  initial^ 
shaping  of  the  strategic  framework  through  which  the  PIC" 
will  achieve  its  programmatic  purpose  and  objectiyes/" 
( 

•   -    \  / 
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3  .     Establish  Initial  Needs<»  Priorities 

•    As  you  kftow,  your  PIC  has  two  broad  categories  of 
programmatic  activities  which  it  can  support: 

•  Training  and  D^velopmpnl-   Prnfrrfg  -  projects 
specifically  directed  toward  CETA  participants 
to  prepare  them  to  secure  unsubsicfized  employ- 

^  ment, 

•  Employment  Generating  Services   (EGS)  *-  projects, 
or  services  not  specifically  directed  toward 
clients,  intended  and  .designed  to  generate  fh- 

. creased  employment  opportunities  in  the  private 
sector  for  the  CETA  participant. 

This^  step  consists  of  taking  the  data  .obtained  in 'the 
needs  assessment  and  reviewing  it  to  establish  a  preliminary 
ranking  of  needs  arid  priorities  within  those  programmatic 
categories.     Establishing  these  priorities  involves: 

•  defining  employment  generating  service 
a               •      selecting  target  groups 

•  determining  occupations  for  training 
programs 

•  ,  defining  participant  training  and  devel- 

opment -service  needs.  $ 

Defining  EGS  NQ^ds 

—  «,  ««■   t 

Under  Employment  Generating  Services,  your  PIC  has  ,the 

responsibility  of: 

•  attempting  to  bridge  the  gap  betweeri  the 
private  and  public  sectors,        •  \^ 

•  helping  to  create  new  job  opportunities 
for  the  un-  and  underemployed 

•  developing  innovative  alternatives  to 

,     traditional  approaches  to  meeting  employ- 
ment *  and  -training  needs. 

To  discharge  these,  responsibilities ,  your  PIC  needs  to 
^^luate  local  circumstances  and  conditions.     By,  using  your 
needs  assessment  .-data  to  answer  questions,  you  can  determine" 
which  needs  are  most  important  at  this  time,  and  tjhen  design  - 
the  Employment  Generating  Services  portion  of  your  planrtb 


address % these  needs.     Questions  you  might  ask  yourself  in 
reviewing  that  data  'are:  . 

•  Is  there^a  lacJ^  of  awareness  of  PSIP? 

•  Is  there  a  negative  impression  toward 
governmental  employment  and  training 
programs? 

•  Do  good  job  6pportunities  exist  within  the 
lOcaJ.  environment  which  go  unfilled  because 
of  ]Jack  of  contact  between  governmentally- , 
sponsored  programs  and  local  business  estab- 
lishments? 

•  Is  there  a  need  to  help  create  jobs  or# to 
develop  the  economic  climate  of  the  community? 

•  Is  there  effective  cooperation  and  communica- ' 
"  tion  among  those  agencies  which  constitute 

your  local  employment  and  training  network? 

•  Have  governmental ly -s£>ojisored  programs  been  . 
designed  to  be  directly  responsive  to  the 
needs  of  lopal  businesses? 

•  What  do  local  employers  »f eel  inhibits  their 
^                  -  employment  of  the  structurally  unemployed? 

•  What  services  can  be  provided  to  enhance  \ 
w  the  l<x;al  business*  communities1  receptivity' 

toward  your  program?        *  . 

Selecting  Target  Groups  • 

To  initiate  the  process  of  setting  priorities  on  th^  train- 
ing and  development  side,  your  PIC  should  select  the  groups  who 
will  be  your  targets  for  both  ,  training,  and  development  projects 
and  services.     The  decision  of  whojn  to  serve  is  very  much  a 
local  option.     You  should  take  the  data  secured  in  the  needs 
assessment  into  account  ia  making  this  decision. 

As  with  occupations ,  you  might  want  to  establish  certain 
factors  or  considerations  to  look  at  in  making  your  determina- 
tion of  target  groups.     Possible  factors  could  be: 

0        relative  overall  severity  of  need  of  each  / 
participant  group 

•  "^mployability "  based  on  characteristics 
(e .g .     e^kjcation,  length  of  unemployment) 
of  the  members  of.  the  participant  group 

•  percentage  of  participant  group  being 
served  by  other  employment  and  training 
programs 
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•    ' ' particular  or  special  needs  of  ' subgroups 
{e.g.,  displaced  homemakers;  ex-offenders; 
alcoholics;  Vietnam  vets). 


Determining  Occupations  for  Training  Programs^ 

As  part  of  deciding  "on  ydur  PIC's.  trainirlg  and  development 
agenda,  you  should  determine ,  *from  among  all  of  those  "growth" 
occupations   (due  to  job  turnover  or  job  expansion)   as  revealed 
by  your  Tailor  market  analysis;  which  should  be  priorities  fo\ 
training  programs  to  be  sponsored  through  your  PIC. 

To  make  this  determination ,  you  should  establish  some 
criteria  that  yau  can  use  to^eview  the  growth  occupations 
and  their  "suitability"  'for  the  participants  in  your  program. 
Possible  fac-tors  for  which  your  PIC  could  establish  specific 
criteria  include:  4  -  % 

•  V  'total  number  of  promoted  openings  for  the 

year 

•  percentage  of  "entry-r  level "  opening 

•  pr£>ject^ed  growth  rate  for  5-year  period 

•  ,  I  hourly^starting  s&^ary 

•  promotion  potential 

•  *    natur*  oX  working  environment  and  Conditions 
.    #  .      skill  >$b&  training  .requirements" 


Defining  Participant  Sfervice  Needs  * 

Irt  addition  to  de-fcermining *the  occupations  which  should 
be  priorities  f6r  trainin^^rograms  to  be  sponsored  through 
your  PIC,   at  thi^  point,   you  should  also  indentify  the  prior- 
ity service  needs  o^thdse  individuals  who  will  constitute  the 
population  for  your  PIC^s  program.     In  certain  instance's,  par- 
tiCipants  may  require  assistance  or  support  in  addition  tcr 
training.     For  example,  a  participant  may  require  health  c<5r£ 

*  or  transportation  assistance.     In  other  instances,   you  may  f ind 
that  target  group  participants  do  not  require  occupational  train 
ing,  bi)t  only  services  such  as  orientation  to  the  world  of  % 
work  or  placement-.     You  can  use  your  needs  assessment  data 

-to  ma)Je  this  determination.     \n  addition-,   the  Prime  Sponsor 
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staff  can  be  a  valuable  resource  for  additional  insights  arid 
advice  irf  this  area. 

Once  your  PIC  has  defined  its  potential  priority  employ- 
ment generating  services,  target  groups,  occupations  for  train- 
ing programs,  and  participant  service  needs,  you  have  the  ba^is 
for  setting  your'PIC's  programmatic  goals. 

9  ' 


4T^P^define  PIC  Mission  and  Functions 

J 

The  CETA  Regulations  spell. out  broad  purposes  and  roles, 
as  follows,   f a  Private  Industry.  Council* 

Purposes :     To  increase  the  involvement  Of  the' 
business  community,  including  small 
business,  minority  business  enter- 
prises, and  labor  organizations  in 
employment 'and  training  activities. 


To  increase  private  sector  employ- 
ment opportunities  for  economically 
disadvantaged  persons. 


Roles:  The  PIC  shall: 


serve  as  an  intermediary  to  assist 
the  local  employment  and  training 
structure  to  become  more  responsive 
to  the  business  community. 

serve  as' the  business  and  industry 
C9ntact  point  in  the  local'  employ- 
ment and  training  system,   to  present 
£he  private  sector's  view  and  recom- 
mendations for  making  programs  more 
responsive  to  local  employment  needs. 

* 

advise  and  provide  direction-  to  £he 
local  employment  and  training  system^ 
on  ways  to  increase  private  sector 
job  placements  for  eligible  persons.^ 


#     Within  this  broad  context, it  is  up  to  your  PIC  to 


define  its  specific  mission   (purpose  and  role(s))  .and'  func- 
tions more  precisely.     In  this  stepv  you  reexamine  your 

c  ■  ..    "  /' 
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present  definition  of  purpose,  role(s),  and  functions,   in  light 
of  the  results  of  your  assessments  of  community  needs  and  PIC 
prd§rammatic  and  operational  performance,  to  determine  whether  , 
they  are  still  appropriate  or  need  to  be  .modified* 

Purpose  . 

\  A  purpose  is  a  statement  of  ultimate  intent  and  results 

expected  in  the  long  term  by  the  PIC-    Your  PIC's  purpose  should 

flow  f rc^tf  'tjfie  Title.  VII  regulations  and  legislation  and  then  be 

aidapted  to  express  your  local  emphasis.  .  Although  the  Title  VII 

5r  * 

regulations  provide  the  initial  mandate  for  your  PIC,  you 
should  looks  at  the  situation  and  needs  within  your  community 
and  jth^* broaden  or  restrict  your  PIC's  scope  to  relate  to 
thos?  specific  needs   (see  discussion  in  Section  I  on, critical 
PME  decisions).   An  example  of  one  PIC's  statement  of  purpose 
Sollows: 

To  increase  substantially  the  involvement  of  f 
the  private  sector  in  publicly-f inancecj  employ- 
rtlent  and  training  programs./ 

To  strengthen  the  capacity  of  the  employment 

and  training  system  to  respond  to  the  needs  of  *  L- 

both  industry"  and  the  economically  disadvantaged. 
%*  • 

To  increase  the  access  to  .good  jobs  for-  the* 
economically  disadvantaged  and  structurally 
unemployed.  "  - 

To  improve  the  local ' investment  climafe  and  to 
increase  coordination  between, employment  and 
training  and  economic  development  programs. 

\ 

Role (s)         %  %  1  ' 

There  is  no  single  prescribed  set  of  roles  for  a  Private 
Industry  Council.  *  The  three  basic  potential  roles  for  a  PIC, 
none  of  which  excludes  either  of  the  others,  are: 

Advisor    -  Reviewer  and  consv^ltant  to  local 
CETA  Prime  Sponsor  in  program  J 
planning,   implementation ,  and  / 
review  process. 

'^Broker      -  Facilitator  and  provider  of 

technical  assistance  fpr  con- 
nections between  employer s, 
the  Prime  Sponsor,  other 


providers  of  employment  and 
•  training  services  and  thje 
community. 

j 

Operator  -  Designer  and  deliverer  of  direct 
services  either  to  participants, 
.employers,  or  other  agencies. 

As  suggested,   the  CETA  Regulations  require  your  PIC  to 
play,  at  least,   the  a^dVisor  or  consultant  role  in  certain  areas. 
Your  responsibility,  at  this  point,  is  to  clarify  the  additional 
role(s)   that  your  Council  wants  to  assume,   if  any,  with  refer- 
ence not  only  to  PSIP,  but  to"  other  employment  and  trailing 
activities  under  CETA  and  related  programs   (economic  develop- 
ment, education,  etc.). 

j 

Functions  •  -  ^ 
 1   i  ^ 

Eased  on  the  role(s)   that  your  Council  selects,  you  should 
determine  the  functions*  that  jou  wish  to  perform.     The  primary 
required  function  for  the  PIC  is  to  design  and  develop  the 
Title  VII  program.     One  other  required  function  discussed 
earlier  is  to  analyze  private  sector  job  opportunities. 

^Examples  of  possible  discretionary  options  as  they  relate 
to  the  three  -basic  roles  for  your  PIC  are  listed  below. 

Advisor 

•  Identify  factors  which  contribute  to  program 
effectiveness 

•  Examine  the  performance  of  CETA  and  the 
Employment. Service 

•  Review  and  comment  on  plans  of  economic 
development  agencies 

Broker 

Between  individual  employers  and  publicly-funded  programs: 

•  pisseminate  CETA  and  PlC  program  information  , 
to  employers 

•  Disseminate  information  on  the  Targeted  Jobs 
Tax  Credit  and  other  employment  incentives 


•  Develop  and. market  model  contracts 


•  Provide  assistance  to  employers  to  reduce 
the  administrative  burden  for  participating 
in  employment  and  training  programs 

•  Encourage  employers # to  develop,  and -to 
coordinate  with  training  agencies,  jab 
skill  requirement  forecasts 

Between  employers  (as  a  group)  and  Qther  groups 
or  institutions  in  the  community: 

•  Coordinate  Title  VII  programs  with 
other  employment  and  training  activi- 
ties' 

•  Develop  linkages*  between  school  and 
work  r 

t  Develop  linkages  between  employment* 
and  training  arid  economic  development 
programs 


Operator 
Programs  for  workers: 

•  Provide  world  of  woi^c  training 

•  Provide  .vocational  skills  training 

•  Provide  entrepreneur ship  training 


Programs  tax  employers: 

*  *■ 

#  Provide  services  linked  to  CBTA 
training  and  placement 

•  Provide  economic  development  and  job 
•creating  related  services 


Programs  for  education/employment  and  training 
staff: 

%  Promote  exposure  to  private  industry 

#  Provide-assistance  in  accomplishing 
specific  tasks 


In  conclusion,  establishing  your  PIC's  mission  a/id  functions 
is  an  essential  precondition  for . establishing  its  *  % 

programmatic,  orientation  and  strategy/      '  ,t 


5.,     Refine  PIC  Organisational 
and  Operational  Framework 

The  PIC  has  a  large  degree  of  latitude  in  establishing 
its  structure,   staffing,%  operational  procedures.,  and 
organizational  status.     The  purpose  of  this  step  is  to  re- 
examine those  basic  working  arrangements  which  have  been% 
established  by  your  PIC  to  see  if  they  are  still  viable. 

Structure  ' 


t  The  size,   composition',  and  structure  of  your  Council 
shQuld  be  dictated  by  local  needs,   interests,  and  the 
jnission  and  function  which  your  PIC  establishes  for  itself. 
Factors  which  you  should  consider  in  reexamining  the  nature 
of  your  PIC  include: 

•  relationships  with  other  CETA  programs 

•  physical  requirements  for  meetings 

•  manageability  of  PIC  'for  staff  and 

chairpersons 

•  workload  requirements  upon  Council 

members 

•  representativeness  cf  Council  in  terms 

of  composition   (Are  small  and  minority  ^ 
businesses  adequately. represented? 
.  #     Does  the  Council  reflect  the  .largest 

possible  segment  of  community  concerns, 
such  as  schools,  economic  development 
agencies?) 

•  staff  and  services  necessary  to  support  / 

the  PIC  ^% 

•  the  membership  Jphould  it  include  Chief 

Executive  Officers  or  other  management 
staff?     This  may^vary  among  PICs  de- 
pending on  their  overall  ro^Le  and  pro- 
grams.) ,  j 
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A  distinct  factor  meriting  special  reconsideration  in 
this  area  is  the  structure  and  design  of  ytfur  standard  sub- 
committees.    PIC  subcommittees  tend  ;to  be  defined  either 
structurally-—i.e.. ,   by  industry:     clerical,  automotive; 

or  functionally^  i.e.,   by  PIC  purpose:     executive,  planning, 

monitoring  anS^eyaluation .  Your  PIC  should  look  at  its  sub- 
committees to  see  if  they:  .  ^ 

•  are  defined  most  appropriately  given 
local  needs 

f 

•  are  useful  for  decision-mATcing  purposes 

>    •    have  proper  membership 

•  * 

•  are  sufficiently  comprehensive  in 


their;  coverage  (Do  you  frequently 
have  to  create  ad  hoc  Committees?) 


\ 


Staffing 

There  are  a  variety  of  possibilities  for  staffing 
PIGs,     ranging  from  part-time  CETA  staff  on  the  one  hand,  to 
full-time  PIC -employees  on  the  other.     Your  need  is  to  deter- 
mine whether  the  staffing  mode  presently  being  used  by  your 
PIC  is  satisfactory.     Other  important  considerations  should 
be  whether  your  PIC's  staffing  level  is  adequate,  and  whether 
the  division  of  labor  among  staff  members1  is  equitable. 

Operational  Procedures  and  Methods 

Another  essential  area  to  reflect  upon  is  your  PtCs 
operational  procedures  and  methods.     The,  assessment  data  J 
will  allow  you  to  determine  Whether  your  PIC's  administrative 
methods  and  processes  for -discharging  the  PlC  s*  business  and 
responsibilities  are  adequate  or  whether  they  need  to  be 
altered.     Administrative  items  that  you  should  look  at  include 

.   •     Council  by-laws  * 
*         •  ^  Meeting  procedures   (formal  and  /  , 

informal  process)  *  ] 

•     Selection  an<3  duties  of  officers 
/         m      Roles  and  responsibilities  of  Council 
members,  staff  \  • 

i  • 
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As  part  of  your  determination  here,  you  should  decide 
whether  your  PICfs  process  for  planning,  management  and  eval- 
uation needs  revision. 

Organizational  Status/ 

4  One  last  important  concern  should  be  your  PICfs  organiza- 
tional statu^.     Again,  as  with  the  other  areas  under  this  head- 
er 

ing,   there  is  no  single  prescribed  PIC  organizational  model. 
Options  which  are  possible  include: 


•  advisory  position  within  CETA  ^Prime  Sponsor's 
H system  *  1 


•  incorporation  as  non-profitf  organization 

•  segment  of  existing  business  organization 
(e.g.,   part  of  Chamber  of  Commerce,  NAB) 

%  multi jurisdictional  PIC  covering  more  than 
one  prime  sponsorship. 

*> 

Your  responsibility  is  to  determine  whether  the  present 
status  of  your  PIC  enables,  it  to  perform  most  effectively  and 
efficiently.  At  a  minimum,  you  should  ensure  that  you  have  a 
direct  and  clear  relationship  to  the  Prime  Sponsor,  the  Prime 
Sponsor's  advisory  council,  and  to  the  chief  elected  official 
in  your  community. 

The  net  result  of  this  step  should  be  that  your  PIC 
institutes  all  alternative  organizational  and  operational 
arrangements  necessary  to  eliminate  those  barriers  which  you 
feel  havS  impeded  effective  performance  to  date. 


>er  to  3 

v 


'  6 .     Formulate  Goals  and  PIC  Strategic ^Plan 

The  final  step  in  this  stage  is  converting  the  resylts  of 
your  assessments  and  analysis,  and  your  PIC's  preliminary  deci- 
sions into^^g^als  and  a  strategic  plan.     A  goal  is  a  broad  un- 
quantified  statement  of  an  outcome  or  result  which  you*  PIC 
wants* to  achieve.     A  strategic  plan  is  a  summary  statement 
which  details  all  of  the  goals  you  want  to  accomplish  in  the - 
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upcoming  year  and  expresses  their  relative  importance,.  The 
plan  also  presents  those  goals  which  are  longer  term  in  nature' 
and  establishes  some  general  time  parameters  for  their  accom- 
'p?ishment. 

Setting  Goals 

There  are.  three  major  areas  in  which  your  PIC  can  set 
.goals: 

•       "Employment  Generating  Services 

m        Participant  Services  -  Training-  and 
Development 

Organizational 

If  you  have  Already  established  priorities  for  the  needs  \ 
employers,  occupations;  and  tarqet  qroups,  and  vqur  PIC  itself, 
and  identified  any  barriers  to 'be  overcome- or  reduced,  then 
establishing  your  PIC's  goals  is  a  straightforward  process 
and  can  almost  be  done  by  formula  as  shown  below: 

Assessment  Data  GoaLs  . 

—  -  %  * 

•    Employer  needs  •+  barriers     m        Employment  Generating 
to  overcome  Services    , % 


Priority  occupations  + 
training  needs  +  target 
groups 

Target  participants  + 
assistant  needs  + 
barriers  to  overcome 


Participant  Services/ 
Training  and  Development 


•    PIC  internal  needs  +  =  Organizational 

s         barriers  to  overconfe 

Example  goals  as  they  Alate  to  each  of  the  above  categories 


f ollow:^ 
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AREA 

NEEDS/BARRIERS 

GOALS' 

Employment 
Generating 
Services  9 

.  Insufficient  labor  market 
date  for  PIC j^pc is ion-making 
purposes . 

.  To  improve  the  quantity,- 
equality,  and  decision-making 
value  of  local  labor  market 
information. 

.  Lack  of  knowledge  in  the 
business  community  regarding 
PIC  programs.            ^  » 

.  To  create  and  implement  *a 
•  comprehensive,  ongoing 

marketing  program  for  PIC 

activities* 

> 

.  Inadequate  relationship 
between  public  and.  private 
settor  economic  development 
efforts. 

< 

.  To  establish  a  coordinated 
long-term  economic  develpp- 
ment  linkage  between  the 
public  and  private  sectors 
with  the  result  of  creating 
greater  job  opportunities  1 
for  economically"  disadvan- 
taged persons. 

Participant 
Snces  - 
Trailing  and 

Development 

.  Large  jmjper  of  unfilled 
clerical,  posit  ions  +  high 
percentage  of  unemployed 
black  and  Hispanic  females 
between  ages  of  18-35. 

.  To  enable  black  and  Hispanic 
fefcales,  ages  18-35,  to 
secure  unsubsi<Jized  clerical 
jobs  in  the  private  seetor. 

* 

.  Lack  of  opportunity  for  up- 
grading within  major  local 
private  sector  industries 
which  employ  signif icant, 
numbers  of  lower  salaried 
individuals  in  entry-level 
positions. 

.  To  institute  effort  with  < 
major'  employers  to  upgrade 
employees  from  entry-level 
jobs  to  positions  of  greater 
skill. 

map 

.  Inadequate  understanding  and 
development  of  general  work 
skills  and  habits  among  - 
youths,  18-21. 

.  To  enhance  the  employability 
of»youths,  ages  18-21. 

Organization 
* 

»  Poor  planning,  management 
and ' evaluation  process  - 

1 1  n  r1 1  oar*    r^of^^^^t"^^^^  of 

UULlCdL     UCi  1111  L  1UII  Ui 

-  responsibilities  and  in- 
adequate system  design. 

.  To  redesign  PIC's  PME 

Process  and  assign  definite 
PMF  resDon<?ib il i t ies  to 
both  staff  and  members. 

\  • 

1  The  ai^as  in  which  you  set  goals  and  the  number  of  goals 

tyhat  you  set  will-  be  determined  by  the  nature  of  your  local  needs. 
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Developing  the  Strategic  Plan  - 

•Setting  goals,  however,  is  riot  the  difficult  part  of 
establishing  your  PIC's  strategic  plan.     The  more  demanding 
task  is  deciding  what  your  PIC 1 s  mix  of  goals  should  be  and 
which  goals'  should  take  precedence. 

This  determination  is  truly  your  option.  The  CETA  regu- 
lations only  stipulate  the  percentages  of  your  Title  WI  funds 
which  can  or  have  to  be  allocated  to  each  category  <?f  activities 

•  Direct  participant  Services.-  at  least  50% 

•  Employment  generating  services  -  up  tp  30% 

•  Administrative  activities--  no  more  than  20% 

•  EcononrfLC  development  activities  -  10%  incentive 
f  or^sponsorship  $ 

Within  -those  guidelines,   it 1 s  up  to  you  to  determine  what  rela- 
tive  priorities,  mix  of  goals,  and  eventual  distribution' of  , 
dollars  can  b^st  meet  the  employment  and  trailing  needs  within 
your  community.     Factors  that  should  be  taken  into  account  in 
making  these  decisions  between  and  among  goals  are: 

•  appropriatenesS( to  your  PIC's  mission 

•  appropriateness  to  economic,  conditions 

•  acceptability  to  community  . 

•  obstacles  to  achievement 

•  trade-offs  in  outcomes  associated  with 
selection  of  one  goal  pver  another.  % 

Finally,  oncje  your  PIC  has  reached  consensus  on  its  goals> 
their  relative  priorities,'  and  the _ rough  percentage  of  the  bud- 
"get--fco.be  allocated  to  each  goal,"  this-inf ormation  should  be 
recorded  as  the  PIC's  strategic  plan  for  the  year. 

•In  addition,   the  plan  should  list  all  other  longer-term 
goals  which  are  important  for  your'  PIC  -and  which  will  be 
addressed  in  succeeding  years.     The  portion  df  .the  strategic- 
plan  relating  to  next  year  provides  the  initial  substantive 
parameters  for  the  full  scope  of  programmatic  activities  to1 
be  undertaken  by  your  PIC. 
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C{    ESTABLISH  >ROGRAM  SCOPE 


RECLAMING  FEEOMCK 


'f 


C.  Establish 
Progrim 
Seopt 


D.  Ottfitp 
Prffrim  Q 

Ptoi 


E*b«i»a  1 

— * 

C.  UlMM 

Krttnm 

MiMiftmut 
PlM 

% 

Purp<$ 


se : 


To  translate  the  broad  qoapf  ^pd  directions 
set  dut  in  the  strategic  plan  into  specific 
program  objectives  and  projects  for  'achieving 
them.  —  ,  x 


v 


Steps ; 


1 
2 
3 


Define  Appropriate  Tactics 

Draft  PIC  Program  Objectives 

Specify  Project^  and  ^Scope 
1+  .  ^ 


Outcome ; ^ 


Use:  ^ 

  i 


A  preliminary  design  of,  PIC's  program  scope 
for  fiscal  year.  .  * 


The  program  scope  is  used  as  the^jpgsis  for 
structuring  the  program's  delivery  system  and 
securing  any  additional  assistance  needed  to 
implement  the  PIC's  program.  ^ 


er|c 
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1.     Define  Appropriate  Tactics 

•  The  first  step  toward  converting  your  PIC ' s  strategic 
plan  into  a  workable  real^^is  defining  what  tactic  (s) 
is  (are)  most  appropriate- and  should  be  used  to  accomplish 
"ehch  goal  that  you  have  set.     A  tactic  is*a  general  means 
or- broadly- jjMfcd  activity  for  achieving  an  end. 

The  rangPl&f 'options  available  to  your  PIC  is  wide.* 
Again,  as  witft  your  goals,  these  options  can  be  broken 
down  into  the  categories  of  participant  services  and  . 
employment  generating  services. 


Participant- Related  Tactics 


v 


Major  participant-related  tactics  include: 

*   #  Classroom  training     (vocational)  -  training 
conducted  away  from  a  job  to  provide  the  - 
'  '     .  participant  with  specific  technical  know- 

ledge and  'skills -necessary  to  perform 
successfully  in  a  job.  /^^^) 

#  Classroop/  training    (  employability)  -  train-* 
¥  -  ;  ing  -eaaducted  away  from  a  job  to  provide 

the  participant  with  the  fundamental  know- 
£  '    ledge,  skills,  abilities,  and  attitudes 

necessary  to  secure  and  hold  an  unsubsidized 
"~  private  ^fector  job.  ^ 

;     4  »  On-the-job^  training  -   training  and,  super- 

vision provided.  $Q  participants^ired  by' 
private  sector^  firms  to  enable  thfem  to, 
develop  all  necessary  occupational' skills 
while  employed  in  a  productive  job  in  the  ,  - 
private  -secj^pr. 

Upgrading  -  skill  training   (either  classroom 
or  OJT)  of  individuals  locked  in  low  paying 
dead-end  jobs  to  enable*  them  to  qualify  for 
positions  requiring  greater}  skill  within  the 
same  business  enterprise.  / 


(  . 


5C 


ERIC 


9 


m  Retraining  -  Skill  training  (either  classroom 
or  OJT)   in  occupational  skill  shortage  areas 
to  teach  participants  new  skills  distinct  from 
those  already* possessed. 

*  Employment  and  training  services   -  services  to 
|        enhance  an  individuals  capabilities  and  oppor- 
tunities for  securing  unsubsidized  employment. 

These  services  include  but  are  not  limited  to: 

-> 

-  orientation  to  world  Of  work 

-  counseling  (employment  related  and 
testing) 

-  eraployability  assessment 

-  job  development 
^       .  -  job  search  assistance 

-  job  referral,  and  placement 

-  Target  ^ffbs  Tax  Credit  -  eligibility 
determination  '  f 

-  vocational  exploration  programs  \ 

•  Supportive^  services  -services  to  eliminate 
possible  barriers  to  successful  empldyment. 
Services  include  but  are  not  limited  to: 

-  health  care  ^d  medical  services 
+  child  care 

-  transportation 

-  temporary  shelter  a 

-  financial  counsel ing^and  assistance"^ 

Employment  GenKratinc?  Tactics 

4    The  tactics  available  to  yoyr  PIC  to  help  create  jobs  under 
the  Employment  Generating  heading  jure  even  more  varied  than  those' 
tactics  which  are  participant-related..    Some  of  the'most  common 
major  tactics  that  your  PIC  mighi^phoose  from  include: 

•  Labor  Market^Analysis   -  studies*  of  local 
businesses  to  identify  the  particular  labor 
market  needs  within  the  community  ^ 

•  Marketing   -  activities  to  expand  awareness,  \ 

•  explain  benefits,   and  provide  assistarxre  to 
loc^l  businesses  to  increase  their  participa- 
tion in  CETA  programs  , 

•  Economic  and  Community^ Development  -  efforts 
to  attract,  retain,  and/or  expand  businesses 
to  enhance  economic  and  community  conditions 
within  the  local  area 
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m  Incentive  /rograms   -  programs  created  to  offer 
or  to  projto.te  specidl  benefits,   such  as  Targeted 
Jobs  Tax  [Credit,'  to  induce  private  sector  employ- 
ers to  h/re  CJffft^ligib^e  participants 


#  Linkage  Ci 


Ipn  -  ef  forts  ^  to'  forge  new 

cisting  institutions'  (i.e.  , 
|es,  business  groups,  com- 
ibor  unions,   local  planning 


relations 
government 

munity  groupie  

bodies,  and  Schools/colleges)   to  improve  . 
employment  and  training  capabilities  within  the 
lQcal  community .  .  m 


Selecting  Your. Tactics  '  ^ 

The  foregoing  list  of  tactics  for  participant  and  ^ 
employment  generating  services  are  meant  to  be  instructive 
and  riot' exhaustive.     These  tactics  are  not.  important 'as 
ends^ln  themselves, .  but  only  as  they  relate  to  the  goals 
you  have  set  for  yourPIC. 

-    To  a  large  degree,  each  of\  your  goals  will  i*ply,  and 
the  analysis' which  preceded- its  development  will  enable 
you  to  select,  a  primary  tactic  Which  w±*3r ^Appropriate 
to  your  end.     The  twQ  examples  below  show  .the  relation 
between  goals  and  barriers,  and  tactics  which  might  be 
selected  in  response  to  them.  ; 


GOAL  1 

To  enable  unemployed 
black  or  Hispanic 
females,  ages  18-35,  to 
secure  unsubsidized 
clerical  jobs  in  the 
private "sector  , 

To  improve  the  quantity, 
quality,  and  decision- 
making value  of  local  - 
labor  market  informa- 
*  tion^ 


•  PRIMARY  BARRIER 
 «  1 

Knowledge  and  skill 

deficiency 


APPROPRIATE  TACTIC (S) 

Classroom  training: 
vocational  - 
clerical  skills 


/ 


Inadequacy  , of  standard  Labor  Market  analysis  ■ 
source  data,       ,  local  employer  survey 


Your  PIC  should  select  a  primary  tactic  to  achieve  each 
goal  that  you  have  set.  '  The  goal  plus  the  tactic  provide  th 
basis''for  setting  your  PIC's  programmatic  object!^  and 
beginning  the  process  of  shaping  yoy 
activities  and  services. 


ir  PIC's  total  program  of 


T 


2 .     Draft  PIC  Program  Objectives 

Once  you  have  goals  and  tactics  related  to  them,  you 
translate  this  information  into  your  PIC's  .program  objec-' 
tives  for  the  fiscal  year.     Program  objectives  tell  you 
what  your  PIC  is  going  to  do  and  how  well . 

A  program  objective  should  be  either  a  quantifiable, 
or  in  some^other  way  measurable,  expression  of  what  is  to 
tfccur  specifically  as  a  result  of  a  program  tactic.  Your 
PIC's  program  objectives  help  to  narrow  the  focus  of  your 
goals  and  provide  the  bridge  between  the  more  abstract  or 
conceptual  strategic  planning  stage  and  the  more  concrete 
stage  of  operational  planning.  They  also 'provide  criteria 
against  which  to  measure  programmatic  performance  at  the 
end  of  the,  fiscal  year.  ~ 

It  is  useful  to  look  at  program  objectives  in  the  two 
categories  described  earlier:  C 

#  Participant-related 

•  Employment  Generating  Services'  related 

(EGS)  •  \ 

The  process  of  setting  objectives  fpr  each  category  is 
similar,   but  the  actual  mechanics  are  slightly  different. 

Participajit-Related  Objectives 

Participant-related  objectives  should ' answer  at  least 
four  ba£ic  questions:  "  . 

Who     -.description  of  people"  b<^l?e  served 

How  Many  -  number  of  people  to  he  served 

What  Means  -  tactic   (primary  service  or 
activity,  to  'beSprovided)    -  w^r- 

Results  -    primary  final  outcome  I 

expected.       .  \  '  * 

4?  An  example  of  a  participant-related  objective  based 
upc^n  a  goal  set  earlier  follows. 
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Goal:     To  enable  unemployed  black  or  "Hispanic 
*  females,  ages  18-435,  to  secure  unsub- 

sidized  clerical  jobs'  in  the  private 
sector.  i 

The  goal  tells  you: 

Who;     black  and  Hispanic  females,  ages  18-35 

Results ;     unsubsidized  clerical  .  jobs  in  the 
private  sector. 

The  tactic  tells  you;  -  _ 

What  Means ;     classroom  training.  ^ 

At  this, point,   it's  up  to  your  PIC  tq  answer  the  question 
of  how  many  in  order  to  create  a  program  objective.     Let,1  s 
say  that  your  Council  answered  that  question  as  follows: 

How  Many ;  7!>. 

JThen,  your  PIC's  program  objectdve  related  to  this  goal 
would  be :  • 

Objective:     To  enable,  through  clarssrdom 
training,   75  unemployed  black  and  Hispanic 
females,   ages  18-35,' to  secure  unsubsidized 
clerical  jolps  in  the  ptivate  sector. 

While  trie  above  is  sufficient  for  a  program  objective/ 
if.  it  chooses,  your  PIC  can  make  its  objective  even  more 
precise  by  answering  other  questions  regarding  results, 
e.g.,  how  well,  at  what  cost,  quality  of  job.     An  ex- 
panded objective  incorporating  answers  tft  IThese  questions 

follows:  4  * 

\      To  enable,  through  classroom  training, 

7  5  unemployed  black  and  'Hispanic  females, 
ages  18-35,  to  secure  unsubsidized  private 
sector  positions  having  an  average  wage 
of  $5 . 25/hj.  at  an  average  cast  pf 
$1 , 500. 00  per  participant. 

Your  PIC  should  make  its  participant-related  program* 
objectives  as  precise  a£  you  feel  they  need  to  be  for  program 
management  purposes.     The  advantage  of  more  detailed  program 


/ 
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objectives  is  that  they  provide  clea-rer  direction  *for  all  plan- 
ning, management,  and  evaluation  activities  to  follow. 

> 

EG£-Related  Objectives 

Given  the  broader  scope  and  somewhat  less  tangible  nature 
(e<.g.,   linkages,  improved  business  relations)   of  Employment 
Generating  Services,  as  opposed  to  those  activities  which  are 
dirfectly  participant-related,   it  will  probably  be  more  diffi- 
cult for  your  PIC  to  s^t  objectives  from  goals  related  to  them. 
"It  is  just  as, necessary  and  important,  however. 

Fortunately,   setting  your  EGS  objectives , can  be  as 
straightforward  a  process  as  setting  your  participant-related 
objectives.     Your  goal  tells  you  the  ge(ne/al  end  you  want  to 
achieve  and  your  tactic  tells  you  What  you  need  to  do  in  order 
tC7  achieve  the*-rend "(e .9.  f  conduct  a  local  employer -survey)  .  - 
To  develop  an  EGS-related  objective,  you  need  only  to  determine 
"how  well."     To  determine  how  well,  you  should  answer  the 
following  questions: 

*  •  By  what  time  or  within  what  time  frame  should  - 

,the  project  be  completed?. 

#  Is  there  er  quantitative  measure  or  value 
which  can  be  assigned  to  the  project  (e.g., 
contacts  to  be  made,  positive  responses  to 
surveys,  reports  to  be  produced,  meetings 

to  .be  held,  new  jobs  filled  by  participants}? 

•  Is  there  a  qualitative  value  which  can  be 
assigned,  to  the  project   (e.g.,  number  of 
acceptable  errors,  adherence  to  some* 
established  external  criterion,  complete- 
ness, acceptability  to  the. Council) ? 

'    •       •  Is  there  a  cost-efficiency/ effectiveness/   *  * 
effort  value  which  can  be  assigned  to  the 
project   (e.g.,  cost  per  report  produced, 
cost  per  person  placed,  number  of  staff 
days  to  be  spent  on  project)? 

Three  examples  of  EGS-related  objectives  pertaining  to  the  goals 
described  earlier  and  answering  all  of- the  aboye  questions 
follow: 


ERIC 


11-39 


6, 


Goal:     To  improve  the  quantity,  quality, 
and  decision-making  value  of  local 
labor  market  information.  . 

Ob jectivei     To*  complete,  within  6  months, 
a  local  employer  survey  'of  100  selected 
leading  industries  in  order  to  identify 
valid  and  reliable  data  Qj}  entry-level 
job  opportunities  and  salary  level  of 
positions  at  a  total  survey  cost  of  no 
more  than  $10, 000. 00. f 


Goal;     To  establish  a  cq^jdinated  long- 
term  economic  development  linkage  f 
between  the  public  and  the  .private    '  « 
sector  with -the  long-term  .result 
of  creating  job  opportunities  for. 
economically  disadvantaged  persons . 

Objectives :   "To"  create  a  joint  committee 
t6  be  composed  of  representatives  from  public 
sector  economic  development  agencies  :and  pri- 
vate sector  investment  firms  $l*dch  w^ll  * 
develop  within  the  next  9  months  a  comprehensive 
local  economic .development  strategy'  at  cost  to 
benefit  CETA-eiigible  participants.     Effort  will 
require  two  person  months  from  PIC  staff. 

To  coordinate  with  the  City  pevelopment 
Department  to  ensure  that  20%  of  construc- 
i     tion  jobs  being  generated  on  the  two  downtown 
redevelopment  -  pro jects ,   supported  primarily 
through  EDA  and  UDAC  funds, are t set  aside  for 
CETA -eligible  part icipants . 

#  ■ 
Again,   your  EGS  objectives  need  not -be  this  detailed. 

At  a  minimum,   however,   for  each  EGS-related  objective  your 
PIC  should  specify  the  end  product  or^  result  it  expects.  £nS 
*  associate  at  least  one  "how  well"  measure   (i.^i  'time,  quan- 
tity,  quality,   cost)   with  it.-  % 

There  are  no  absolute  numbers  of  program  objectives 
that  ought  to  be  set  by  any  PIC. ■   Th£  essential  considera- 
tion should  be,  as  noted  earlier,  that  your  PIC  set  at  least 
on«r objective  for  each  goal/tactic  that  you  intend  to  under- 
take. 
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3 •     Specify  Projects'  and  Scope 

The  last  step  in  the  development  of  the  preliminary 
design  of  your  PIC's  program  for  the  fiscal  year  is  to  iden- 
tify those  specific  projects  to  be  undertaken,  •  Projects  are 
the  discrete  services  and  activities  to  be  sponsored  through 
the  PIC  in  order  to  achieve  its  programmatic  objectives. 
There  are  thrqe  tasks  involved  in  project  specification: 

•  Determine  number  and  types  of  projects  > 
#\Define  objectives  and  scope  of  each  project 

•  Allocate  resources  to  projects 

.  An  example  of  a  program  objective  for  whigh  you  would 
^  proba-bly  have  only  one  project  is"  the  EGS  objective  related 
to  labor  market  analysis: 


Objective;     To  complete,  within  6  months, 
a  local  employer  survey  of  100  Selected 
leading  industries  in  o^cler  to -identify 
valid  and  reliable  data  on  entry-level 
job  opportunities  and  salary  level 

positions  at  a  total  survey  cost  of  no  ► 
more  than  $10,000,00. 

The  sole  p^bject  to  be  undertaken  with  reference  to  this 

objective  would  be  a 'local  employer  survey. 

»  % 

On  the  other  hand,  there  are  other  areas  in.  wj^ph  your  % 
PIC  might  initiate  multiple  projects.  *  For  example, 
referring  back  to  the  participant-related  objective  for 
clerical  positions^    _    '   

r 

To  enable,  through  classroom  training, 

75  unemployed  black  and  Hispanic 

females,  ages  18-35,  to  secure"  unsub- 

sidized  private  sector  positions  having 

an  average  wage  of  $5 . 25/hr .  at  an 

average  co£t  of  $1 , 500. 00 .pgr  partici- 

panVi  ,  » 

It  is  possible  that  your  PIC  might  want  to  split  up  this 
objective  into  two  projects  or  more  with  each  project  being 
Operated  by  a  different  service  deliverer   (e;g.,  community  . 
college;   private  sector  proprietary  institution,  non- 
profit community-based  organization) . 
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As  another  example  of  a  program  objective  which  would  be 
met  through  multiple  pro jects , , let ' s  look  at  an  objective  re- 
lated to  an  example  goal  stated  earlier  in  the  text: 

Goal f    To  institute  a  program  with  major  local 

employers  to  upgrade  employees  from  entry- 
level  jobs  to  positions  of  greater  skill 
and  higher  salaries. 

£  *       Objective:  „  To  institute  a  program  with  major 

local  employers  to  upgrade  at  least  200  employees 
*        from  entry-level  ]ob3  to  positions  of  greater 
skills  and  higher  salaries. 

Determining  Project  Number  and  Types  , 

Projects  are  .derived  directly  from  programmatic  objectives. 
In  some  instances,   you  may  have  only  one  project  per  objective. 
In  others,   you- may  have  as  many  as  three  or  four  projects .  This 
fact  is  represented  in  the  flow  chart  below  which  summarises 
PIC  activity  to  this  point  in  decision-making  terms: 

PIC  Planning  -  Decision-Making  Flow  Chart 
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*  Obviously,  this  objective  calls  for  the  designation  of 
a  specific  number  of^ employees  by  name  and  participants  per 
employer  for  the  upgrading  program  as  an  integral  par*  of 
the  planning  process. 

Setting  Scope  and  Objectives 

Once  tile  number  and  type  of  projects  for  each  objec- 
tive  are  clarified,-  your  next  task 'is  to  establish  the 
specific  scope  and  objectives  for  >each  project.  Essen-' 
tially,  this  consists  of  breaking  the  program  objectives 
down  and  assigning  specific  objectives  to  each  project. 
This  is  done  by  answering  the  same  questions  posed  earlier 
to  establish  your  PIC's  program- objectives  related  to  ] 
Participant  and  Employment  Generating  Services. 

Participant^  Employment  Generating  Services 

Who?  Primary  Result?  * 

How  Many?  ■  ,   How  Well? 

What  Means?  -  Time? 

^    _        «      -.^  "  Quantity? 

*     Primary  Result?  *      _  Quality? 

How*  Well?  -  £bst? 

The  end  result  of  this  process  should  be  &  clear  state- 
mant  for  each  ptoject  of:  . 

•  principal  outcome  immeasurable  terms 

•  primary  service  or  assistance  to  be  delivered 

•  other  activities  and  services  that  comprise  the  % 
project. 

s 

Allocating  Financial  Resources 

After  all  projects  are  thoroughly  defined,  your  PIC's 
remaining  task  is  to  allocate  the  financial  resources 
necessary  to  accomplish  each  project.     As  discussed  earlier, 
the  initial  decision* regarding  fund  allocations  is  made  when  ^ 
you  develop  your  strategic  plan.     At  that  time,  you  obligate 
.the  approximate  amount  of  dollars  t^aat  you  want  to  spend  to  achieve 
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each  goal.'    This  decision  is  checked  again  when  you  set 
your  PIC  1 s* program  objectives. 

v/       Now,  as  the  final  act^rior  to  , developing  your  plan 
for  operation  of  the  PIC's  program,  you -break  out  the 
amount  'to  be  expended  on  each  projedt  in  order  to  estimate 
cost  feasibility  and  to  ensure  that  there  are  sufficient 
resources  to  deliver' the  full  scope  of  program  services 
that  are  planned.     If  the,re  are  shortfalls  either  overall 
or  within  a  given  area,  you  rebudge»t  and  adjust  ^our  pro- 
gram objectives  and  project  objectives  and  scope  accordingly 
After  this  activity  is  completed,  you  are  in  a  position  to 
develop  your  PIC's  plan  for  program  operations. 
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D.     DEVELOP  PROGRAM  OPERATIONS  PLAN 


-r  *MU«HUIC  feeoiacx 
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/ 
/ 
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Purpose: 


Steps : 


To  establish  a  detailed  plan  and  procedures 
for  .operating  your  PIC's  program. 


1. 
2, 
3. 
4. 


Structure-  Program  Delivery  System 
Establish  Performance  Standards 
Secure  Necessary  Assistance 
Finalize  and  Approve  O^^ations  Plan 


Outcome : 


A  written  statement  which  spells  ott  clearly 
your  PIC's  performance  standards,  and*its 
specific  agenda -and  methods  for  addressing  the 
employment  and'  training  needs  of  the  community. 


Use :  The  fProcjram  Plan  serves  as  the  ^primary  guide  and 

reference  for  PIC  program  operations.     Used  in 
conjunction  v/ith -your  PIC's  Management  Plan  and 
Evaluation  Plan,   it  constitutes  a  comprehensive 
framework  fot'  ensuring  satisfactory  program  imple- 
mentation aod  performance. 


9 
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1.     Structure  Program  Delivery  System 

In  this  step,  your  PIC  makes  the  decisions  which  are 
probably  most  critical  to  its  programmatic  success  or  failure. 

^  These  decisions  relate  to  how  you'  are  going  to  deliver  what  you 

have  deeidelfupon  as  your  program 'for  the  year.     Basic  consider- 

-  ■  ations  to.  be  factored  into  your  decision-making  at  this  point 

include:  - 

•  what  parte  of  the  Prime  Sponsor  delivery  ' 
system  you  will  want  to  us| 

^  *  *  how  much  direct  control  your  PIC>iU.l  want 

over  those  projects  which* you  sponsor;  what 
the  PIC  wants' to  operate  directly 
what  i£ "wants  others  to  provide 

•  what  the  relative  tradeoffs  are  in  terms 
of  cost  and  credibility  among  various 
types  of  .service  deliverers m 

•  should  .essential  services  be  delivered  on 
a  decentralized  or  centralized  basis. 

There  are  three  primary  tasks  involved  in  structuring 
•the  delivery  system: 

•  Designing 

9  Developing  Project  Delivery  Framework 

•  Establishing  Administrative"  Infrastructure 

Project  Design 

.         .  .— -      The  first  order  of  business  in  establishing  your  PIC's 

T?"  delivery  system  is  designing  the  projects  which  are  to  be 

initiated.     If  your  PIC  has  decided  to  run  all  or  certain 
projects' itself ,  then  a  great  amount  of  your  attention'  and 
effort  .will  go  into  this  area.     However,,  syice  the  majority  of 
PICs  contract  most  activities  at  this  time,  basic  princi- 
/         pies  and  factors  related  to  effective  project  design  are  dis- 
cussed under  "Step  3.  Secure  Necessar^Cssi stance"  which  follows. 
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Project  D^i very  t  Framework  - 

f         Regardless  of  whether  *  your  PIC  implements  all  its  own 
projects  /(^contracts  them,  or  is  somewhere  in  between,  you  need 
*  to  -establish  a*n  overall  framework  for  ^coordinating  and  inte- 
grating ^pjor  PIC's  projects.     That  framework  will  have,  two 
aspects. —   (1;  training  and  development  or  participant  acti-, 
vities  and   (2)   EGS  activities,  ♦ 


a  . 


^         Training  and* Development   (Participant)  Activities 

Tie  general  model  for  CETA  parfccipant  client  flow  is 

indicated  in  the  following  chart.  -  J ^  — 

*  **• 

~CETA  Client  Flow  ModeT\  * 


Outreach/ 
Recruitment 


Intake/ 
Assessment- 


Training  or 
Development 


Placement/ 
Termination 


Follow-up 


Participant 
Services 


y 


/ 


11-47 


( 


6:) 


*  1  /~ 

Basic  questions  that  you  can  ask'  yourself  as /they  relate 
to  the  foregoing  model  and  your  PIC's  projects  ir/clude: 

•  +  How  will  jrour  PIC  handle  "its  outreach, 

recruitment,   intake,  anpl  assessment  of  ^ 
participants?     Through" a  cen£ral\spurce? 
Through  -each  service  deliverer? 

#  What  pro^isJtans  wall  be  made,   if  any,  to     .     ,  r 
meet  the  spe^al  development  and  support 
service  needs  of  participants? 

9  Will  any  placement/}  job  development,  and 
program  termination? activities  be  under- 
taken? | 

*    •       •  Will  your"  PIC  .inakel  special  arrangements 
for  follow-up  assiptarrtfe  to  program  ser- 
'   vices?    Yes^  No? 


Your  answers  to  these  and\other  questions  which  you 

\  { 
consider  important  will  determine  the  sjiape  of  the  partici- 

apt  service  aspects  of  your  PIC\s  delivery,  system. 


papt 


EGS  Activities 


To  structure  the  EGS  side' of  your  delivery  system,  you 
Should  answer  questions  including: 

m  What  end  product^  or  results  will  be 
produced?    During  what  time  period  or 
by^/hat  date? 

»  What  interim  products  will  be  produced? 
0     By  what  date? 

'  *" 
m  .What  major  activities  will  be  accom- 
plished?    By  what  date? 


lat  are.  project  benchmarks?    When  will 
they  occur?  *  -  /  # 

.  ¥  '  *  r< 

*  What  are  the  major  tasks  or  steps  neces- 
sary to  complete  each  project?  What  .are 
their  timelines? 
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Answering  these  questions  for  all  EGS  proje^s  will 
enable  you  to  develpp  a  time-phased  produfct/actrvj^y 
schedule  which  can* be  used  to  direct  the  implementation 
of  you*  PIC's  EGS  projects  over  the  course  of  the -year. 


Administrative  Infrastructure  * 
 *  «t —  

As  the  l^st  task  in  developing  your  program^de'livery 
system,  your  PIC  should  design  and  put  into  place  an  ad-  y 
ministrative *  infrastructure'  which  complements  and  supports 
your  decisions  regarding  participant  and  EGS  project  deli- 
very.    To  do  this,  you  must  review  and/or  decide: 

« 

•  What  projects  the  PIC  will  run  itself 
and  the  staffirfg  required' 

•  How  to  direct,  coordinate,  and  control 
all  projects  to  be  initiated 

•  How  to  evaluate  the  net"  results  of  your 
m  projects 

•  Who  will  be  responsible  'for  what  . 

je  decisions  that  you  reagh  her-e  will  determine  the 
d'kyM^o-day '  administrative  organization*  and  the  staffing 
needs  for  your  PIC.  \In  addition,  they  will  have  an  impact 
on  your  PIC '  sv  Management  and  Evaluation  Plans  '  (see  Sections 
E  ari^  F~ following) . 

v 

Establish  Progr^amt^  Per f  ormance  Standards 
Afteir  you  have  structured  your  program  delivery  system, 
y6ur  PIC's  next  step  is  to  establish  its.  program  performance 
standards  for  the *f ical  year.     Performance  standards  are 
measurable  descriptions  of  what'  constitutes  a  job  well  done 
far  yoyir  PIC.  ^ 

Performance  standards  are  useful  in  .all  phases  of  the 
PIC  *  s.  program.  .They 

•  -    help  you  to  target  resource's  and     x  X 

activities  during  -the  planning  ( 
process  •  * 

•  serve  as  measures  .which  can  be  used  . 
on  an  ongoing  basis  to  asses  relative 
progress,  and  performance  ^  *" 

provide  a*  means  for  making  an  objec- 
»    '  tive  assessment  of  PIC  program  performance .  _ 

.  . 1      '      ■  f 
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If  you  set  program  objectives  as  described  in  Stage  C 
"Establish  Program  Scope,"  then  you  already  have  the^p^imary 
in£ut  to  use  in  establishing  your  PlC's  performance  standards. 
To  make  .these  objectives  into  standards,  ydu  adjust  them  to  re- 
flect any  new  insights  which  you  may  have  attained  as  a  result 
of  structuring  your  delivery  system  o^r  to  account  for  changed 
conditions  in  the  local  labor  market  or  economy.     These  stan- 
dards will  then  relate  directly  to  your  PIC ' s  goal  and  tactics. 
They  should  be  sufficient  for  program  management  and  review 
""purposes.      '  *  . 

i    In  addition,   at  this  point, you  may  want  to: 

•  refine  the  standards  y©u  have  developbd 

•  create  additional  overall  standards  -  this 

is  probably  most  true  for  the  participant  v 
.    f  service  side  of  your  program. 

The  discussion  below  describes  various  indicators  that 
can  be  use4  to  develop  Standards  for  the  participant,  employ- 
ment generating  and  administrative  sides  of  your  program. 

Indicators  tell  you  what  is  to  be  measured.     They  are 

  4$ 

measurable. "factors  indicating  effectiveness  which  relate  to 
a  certain  area  of  activity. .   By  establishing  a  specific  * 
acceptable  value  for  each  indicator  that  your  PIC  selects, 
you  can  create  a  program  performance  standard.  ~- 

Training  and  Development   (Participant  Service)  Standards 

The  Department  of  Labor  i-s  presently  developing  national 
'  performance  standards  for  application  to  all  CEfA  Prime  Spon- 
sors.   | Indicators  which  have  been  selected  as  appropriate  for 
Titles  II-B  and  II'-D  —  those  CETA  program  components  closest  * 
to  Title- VII 's  in  intent  'and  structure  —  are: 

W%  entered  employment  rate 

* „  •  positive  termination  rate 

ji  average  wage  at  placement 

•  cost  per  entered  employment 

•  cost  per  positive* termination. 

Your  PiC  ma/  want, to  establish  its  own  standards  for  each  of 
'    these  indicators.     Additional  overall  participant  indicators 
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..for  which  your  PIC  might  develop  standards,  or  ftbjectives 
^include:  - 

indirect  placement  rate 
private  sector  placement 
cost  per  indirect  placement 
total  cumulative  enrollment 
tot^l  current  enrollment 
total  accrued  expenditures. 

Examples^  of  participant-related  performance  standards  estab- 
lished by  two  PICs  follow: 


PIC  A  -  Performance  Standards  for  All  Classroom 

Training  Projects 


ERLC 
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Assessment  category 

Indicator 

Standard 

TERMINATION 

Positive  TerBiinationfcRate 
Entered  Employment 

77.1% 
68.0% 

COSTS 

Cost  per  Positive .Termination 
Cost  per  Entered  Employment 

$4,856.00 
5,511.00 

RESOURCE  USE 

Total  Cumulative  Enrollment 
Total  Current  Enrollment 
Total  Accrued  Expenditures 

\         170  * 

J? 

$490,468.00 

PTP  R  -  Peri 

 :  3   1 

•prrn^np  Standards  for  Various 

Projects 

  — ^  , 

■  c 

Projects 

p  — 1                             •           +  X 

Indicator    '  \ 

Standard 

OJT 

Entered  Employment  ^^^-^ 

Satisfactory  Completion  and 
Retention  by  Employer 

70% 

,  65% 

CIASSROOM 
TRAINING 

Entered  Employment 

%  Entered  Employment  in 
Private  Sector  Jobs 

Positive  Termination 

 — >70%  *  

90% 

1  75% 

PARTICIPANT 
PLACEMENT 

%  of  Non-Administrative  Funds f 

to  be  Expended  as  Transfer/ 
^ Payments  to  Participants 

70% 

minimum 
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73 

4 

Other  indicators  for  which  your  PIC  <could  develop  stan- 
dartte  *s  they  relate  to  more  specific  parts  of  the  participant 
activities  of  your  program  include:  %  * 


Component 
Outreach  and  Recruitment 

Intake 

Vocational  Exploration 
Upgrading 

Retraining 


Job  Development 


Possible  Indicator 

#  %  of  eligible  referrals  in 
relation  to  planned  referrals 

*  %  of  applicants  confirmed 
eligible 

#  %  of  successful  completers 

•  Number  or  %  of  participants 
promoted 

*  Number  or  .%  of  participants 
securing  new  positions  with 
salary  equal  or  greater  to 
previous  wages  t  ■ 

*  Number  of  jobs  developed 


Employment  Generating  Service  Standards 

Indicators  which  your  PIC  might  use  in  developing  stan- 
dards for  employment  generating  services  are  preseftted  below 
by  the  major  category  to  which  they  relate: 


Category 
Linkages 


Possible  Indicators 

Numbed  ,pf  formal  or  informal 
agreements  for  loaned  staffV 
use  of  facilities  and  other 
resources^ 

•  Number  eft  formal  or  informal 
agreements  completed  with: 

-  businesses 

-  community  groups 

-  unions 

-  schools 

-  others 

•  Number  of  business  schools 
involved  in  joint  training 
programs  with  PIC 
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Category 

Economic  and 

Community 

Development 


r 


Incentive  . 
Program 


Marketing 


Possible  Indicators 

+     Number  of  jobs  retained   (or  added) 
locally  as  a  result  of  issuing  in- 
dustrial revenue  bonds  to  companiesj 
about  to  close  or  relocate . 

•  +  Number  of  new  jobs  created/ear-marked 

for  PIC  participants, 

•  Number  of  new  jobs  ^filled  by  PIC 
participants . 

•  Number  of  new  jobs  for  PIC*  participants 
developed  as  a  result  of  attracting  new 
business  ' 

•  Number  oT  program  participants  success- 
ful in  getting  funding  and  'training  for 
opening  a  small  business.      '  - 

•  Number  Qf  new. small  businesses  estab- 
lished in  target  neighborhoods.  * 

•  Number  of  workers   (or  program  partici- 
pants)   served  by  PIC-initiated  trans- 
portation program. 

•  Number  of  program  participants  who 
have  found  permanent  employment  in 
PIC-initiated  day  care  centers. 

•  Number  of  program  participants  placed 

in  jobs  who  are  also  using  PIC-initiated, 
'  *   ^day  care  centers. 

•  Number  of  employers  hiring  participants. 

•  Number  of  participants  placed  through  . 
use  of  TJTC  or  other  incentive . 

•  TJTC  positions  generated  through  seminars 
and/or  direct  sales  efforts . 

•  v Percentage^or  number  of  new  businesses 

participating  in  the  hiring  or  training 
of  'PIC  participants. 

* 

m     Number  of  telephone  and/or  letter  

inquiries  from  employers  as  result  of 
specific  advertising  effor^ 
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Administrative  Standards 

The  last  major  area  irx  which  your  PIC  might  want  to 
establish  pfogram  performance  standards  is  on  the  general 
administration  of  the  PIC  and  PSIP  itself.     Using  indicators' 
such  as: 

•  representativeness  of  Council 

ft  average  attendance  at  Council  meeting 

•  description  of  operating  policy  gjid 
procedure 

p  adequacy  and  accuracy  of  program  accounting 

•  .   f requency* and  quality  of  reports, 

yout  PIC  can. create  internal  performance  standards  which  will 
contribute  to 'overall  program  effectiveness.  ^' 
In  conclusion',  a  caution  is  in  order,  as  the  foregoing 
discussion  suggests  it  is  possible  to  establish  performance 
standards  for  almost  everything  and  anything-     You  should  not 
attempt  to  set  program  performance  standards  for  every  area 
of  PIC  activity.     Father,  your  PIC  should  restrict  its  program 
.standards  to  those  areas^and  indicators  which  you  feel  are  key 
or  critical  to  assessing  your  PICfs  program  gerformance. 


3 .     Secure  Necessary.  Assistance 

The  determination  of  performance  standards  is  one  of 
many  considerations  in^eciding  whether  the  PIC  should  operate 
programs  directly  oiL-seek  outside  support.     Obviously,  your 
staff  and  resources  will  help  dictate  the  limits  of  your  capa- 
bilities,  but  performance  standards  can  indicate  to  you 
whether  there  are  existing  deliverers  that  can  meet  your 
requirements,  or  whether  other  deliverers,   including  yourself 
anAbusiness,  should  be  considered.     The  extent  to  wbich  your 

9 

PIC  decides  to  secure  outside  assistance  will  be  determined  by 
the  basic  approach  that  it  takes -to  delivering  services.  If 
your  PIC  needs  outside  assistance,  the  process  that  your  PIC 
should  follow  to  obtain  that  assistance  is  described  or;  the 
followplg  pages. 


11-54 


76 


^  :  

Create  List>(s)   of  Potential  Service  Providers 

A  necessary  pre-step  to  securing  assistance  is  creating 
a  list  of  potential  service  providers  for  each  project  which 
is  tp  be  contracted  out.     Your  PIC  can  draw  these  lists,  in 
part,   from  the  inventory  maintained  by  the  CETA  Prime  Sponsor. 

The  Prime  Sponsor's  inventory  will  probably  consist 
mainly  of  agencies  or  entities  which  currently  provide  or 
have  provided  services  to  CETA.     As  part  or  your  responsi- 
bility to  increase  the  private  sector !s  involvement  in  CETA 
programs,  you  should^ expand  the  list  to  include  businesses 
and  business  organizations  with  the  capability  to  provide 
services  which  are  the  same  or  Amilar  in  nature  to  those 
desired.     You  should  attempt  to  make  your  potential  "bidders" 
list(s)   as  complete  as  possible,   since  the  list  determines 
what  organizations  will  -receive  Requests  for  Proposals  (RFPs) 
issued  by  the  PIC.  .  •  * 


Develop  and  Issue  RFPs 

"Your  PIC  should  develop   and  issue  a  Request  for  Proposal 
for  each  project  to  be  funded  through  the  PIC  but  to  be  operated 
by  an  external  agentf.     The  amount  of  detail  in  each  RFP  will 
vary  from  project  to  project,  depending  upon  the  experience  of 
the  recipients  and  whether  or  not  competitive  procurements  are 
being  taken. 

At  a  minimum,   however,  each  RFP  should  consist  of: 

•  information  regarding  the  PIC  and  PSIP 

•  objectives    of  the  project 

•  ^    detailed  statement  of  the  scope  of  work 

(assistance/services)  desired 
*•     performance  measures'  and  standards  to 
be  used  in  assessing  projects 

•  performance  levels  of  prior  programs 

or  projects 

;   •s    criteria  by  which  the  RFP  will  be  evaluated- 
'(see  next  heading- under  this  step) 

•  performance  schedule  ,  • 

'        reporting  and  management  requirements 

description  of  the  procurement  policies 
^    and  regulations"  of  the  PIC, 

*. 
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Review  'and  Select  Service  Provide^ 

.  ,In  order  to  ^elect  a  successful  proposer,  there  are  some, 
general  considerations  which  you  should  take  into  account  in 
reviewing  each,  proposal  submitted  in  response  to  an  RFP. 
They  include: 

•  proposer's  understanding  o'f^  scope  of  >work 

•  •'       sufficiency  of  content  an^  work  description 

•  *  organizational  capabilities 

related  experience  ^ 

•  relative  costs  for  delivery  of  services 
in  comparison  to  other  proposers . 


*  Title  VII  Proposal  Review  Criteria 

An  example  of  an  in-depth  set  of  proposal  review  criteria 
developed  by  a  PIC  for  a  participant-related  activity  is  pro- 
vided below: 

Program  Impact/Outcome  -  Potential  contribution  of 
the  program  toward  the  resolution  of  the  problems 
experienced  by  the  eligible  populatic^n,  and  the  _ 
program's  potential  contribution  toward  the  attain- 
ment of  PIC  goals. 

•  Will  the  proposed  program  result  in  placements 
in  the  private  sector  at  a  rate  acceptable  to 
the.  PIC? 

•  Will  the  .proposed  program  result  in  proportional 
placement  in  jobs  for  all  significant  segments? 

•  Will  the  proposed  program  facilitate  and  result 

in*  retention  of  participants  by  employers  after  # 
placement? 

•  Will  the  proposed  .training  and/or  services 
enhance  the  upward  mobility  and  promotional 

mj     opportunities  of  participants? 

I  . 

Program  Design  -  Potential  success  of  the  program 
design  in  the  attainment  of  the  described  outcomes/ 
impact.  * 

•  Does  the  proposed  program  satisfy  one  «r  more 
of  the  criteria  for  projects  to  be  funded  under 
Title  VII  within  the  PIC? 


t 
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•  Does  the  proposal  contain  a  feasible  design 
and  rationale  describing  who  will  do  what, 
when,  where  and  ho^  (i.e.,  program  activities 
and  participant  flow)?  ^  ^. 

•  If  -the  cooperation  of  other  organizations  is 
necessary  to  successfully  implement  the  program, 
does  the  proposal  describe#  the  coordination  with 
those  other  organizations?  * 

•  Does  Vtre^proposal  describe  adequate  and  poten- 
^  tially  successful  procedures  for  outreach,  re-  ' 

cruitment  and  assessment  of  participants  that 
will  ensure  enrollment  of  significant  segments 
at  the  proposed  rate£? 

•  *  Does  the  proposal  adequately  describe  training 

for  occupations  which  are t  in  demand  and  explain 
how  persons  trained  will  be  successfully  placed 
into  such  jobs? 

•  Does  the#  proposal  adequately  describe  how  the 
offeror  will  provide  the  support  necessary  to 
sustain  participants  throughout  the  training 

,       in  order  to  reduce  attrition? 


Program  Cost  -  How  the  following  cost  factors 
compare  with  competing  proposals. 

•  Cost  per  participant  enrolled  (total  program 
cost  divided  by  the  jiumber  of  participants)  . 

•  Cost  per  placement   (total  program  cost 
divided  by i the  number  of  participants  placed). 

'  •      Total  costs   (total  program  costs  minus  income 
transferred  to  participants) . 

«•      Does  the  proposal  adequately  explain  and 
justify  the  costs  of  the  program? 

Management  Plan  -  Ability  of  the  organization  to 
implement  the  proposed  program  and  attain  the  stated 
objectives . 

•  Does  the  proposer's  past  relevant  experience 
indicate  a  potential  for  successful  management 
and  implementation  of  the  proposed  program? 

•  Does  the  proposer  demonstrate  success,  or 
reflect  a  potential  for  success  in  providing 
the  described  services  and/or  training^ 


H-57 


7'J 


Does  the  proposer  demonstrate  success,  or 
reflect  a  potential  for  success,  in  serving 
the  Eligible  population? 

Does  the  management  plan  adequately  and 
feasibly  describe  the  necessary  activities, 
events  and  timeline  for  successful  implemen- 
tation of  the  program? 


v  Reviewing  Current  Deliverers 

The  CETA  regulations  require  that  your  PIC  give  special 
consideration  to  those  current  deliverers  or  project  operators 
of  "demonstrated  effectiveness."     The  concept  of  demonstrated 
effectiveness  is  left  open  t^o  local  interpretation. 

Actions  which  PICs  have  taken  to  give  substance  to  the 
concept  include: 

•  assessing  compliance  with  CETA  regulations 

•  evaluating  planned* vs.  actual  performance  - 
(15%  deviation  indicating  potential  problems) 

•  comparing  the  per f ormance^f  individual  vendors 
•  in  the  same  category   (e.g.,  classroom  training) 

•  comparing  performance  of  'individual  vendors  to 
performance  standards 

"*One  PlC^established  the  following  guidelines  to  review 
the  competence  and  projected  effectiveness  of  existing  train- 
ing deliverer^:  4 

Is  training  entity  well-established? 

Does  training  entity  have  adequate  facilities? 

Will  'it  provide  counseling  and  placement 
services?* 

Has  the  curriculum  been  developed  with  private 
sector  input?  v 

Does  entity  have  adequate  accounting  procedures? 

What  are  criteria  for^instructors? 

Will  attitudinal  and  skill  "training  be  provided? 

Will  training  qualify  CETA  participants  for  job 
*  openings2 

What  are  procedures  for  placing  workers  'in  un- 
isubsidized  >  jobs? 
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PICs  £re  also  in  a  unique  positioi)  to  identify'- businesses 
as  deliverers.  Many  businesses,  particularly  larger  ones,  have 
excellent  training  facilities,  staff  and  curricula;  '* 

Your  PIC  should  develop  and  utilize  its  own  set  of 
criteria  to  sectlre,  the  best  service  deliverer  possible  from    »  4 
among  those  respondents  to  your  RFP .  ^ 

Negotiate  and  Develop  Contracts 

-    After  you  have  identified  the  best  proposer,  you  should 
negotiate  a  contract  or  agreement.     This  process  should  include 
clarifying  any  points  you  have  regarding  the  RFP  or  areas  in  . 
Wfiich  you  want  to  see  the  proposal- modified.     Once  this  process 
is  completed,  you  should  develop  the  formal  contract.     To  the 
extent  possible,   thfs  contract  should  follow  the  format  of  the 
RFP  and  incorporate  the  proposer's  responses  to  the, RFP.  The 
oontra'ct  should  describe: 

•  Program  description/design 

•  Goals  and  objectives  • 

•  Outcomes  -  quantitative  performance  standard^ 
qualitative  performance  standards  against 
which  to  evaluate  success  ^ 

•  How  project  will  fit'in  with  overall  PIC 

•  operations 

•  Target  groups   (if  training  and  development 
agreement)  . 

•  Component  design  and  participant  flow 

•  Contractor  responsibilities  -  reporting 
requirements,  program  timeframes,  require- 
ments to  agree  to  monitoring,  audits,  etc., 
and  CETA  Act  and  regulations,  additional 
requirements  and  responsibilities  as  necessary 

•  Contract  payment  process 

.    •  Participant  payment  process   (if  training 
and  development  agreement). 

;     ■  ■  :  ' 
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Having  e-stablished  your  program  performance  standards,  k 
your  last  responsibility  prior  to  actual  program  ^mplenjenta-  ■  * 
tion  isf  to  finalize  and  approve  the  PIC's  program  .opexati oris  ' 
plan.     The  primary  task  here  is  to  prepare  work  plans  or  «t>ate- 
ments  to  guide  all  ^f^cets  of  the  PIC's  operation  and  to  develop 
your.1  final  detailed  cost  estimates.     After  this  is  accomplished, 
the  Council  shoulci  approve  the  entire  operations  plan.  * 

Develop  Woipk  Plans 

Work  plans  ?re  like  road  maps.     They  provide  a  detailed 

picture  of  the  work  to  be  done,  procedures  for  doing  it,  and 

p. 

precise. and  measurable  standard's  of  performance  ,for  each  acti- 
vity.  which  is  undertaken.     Your  PIC  should  develop  or  have  work  | 
plans  for  all  projects  and  major  areas  of  activity   (i.e.,  admin- 
istration-, management  and  evaluation)  participant*  services ,  „ 
training  and,  development  services employment  generating  Services)  \ 
"» '       Work  plan£  normally  describe :  ,  '  ( 

t      •     services  to  ,be  provided/activities  to  be  ' 

'accomplished  ^  #  ) 

-  *     planned  inputs  and  outputs  in  terms  of 

monthly  or  regularly  scheduled  objectives 

•  work  to  be  performed  by  individuals  and 

-  unrtg  (responsibility  assignment)    .  :  • 

•  organizational  charts  ar^d  job  descriptions 
for  individuals  wotking  on  project  or  in 

areV.    *  "  '  "  .  ■  '  "  ' 

•  (   schedule  and  action  steps  for  accomplishing 

work-  ' 

-  •     administrative  procedures  to  be  followed 

•  «v  monitoring,  arrangements "  including  frequency  , 
*  and  checkpoints.'  /  9 

>  .  '         -  1  '  '       ^  \ 

Final  Costs  .  '  /  ' 

As  part  of  putting  together    y6ur  PIC's  operational  plan, 
you  need  to. do, one  more  check  on  coats  to  be  certain  they  are 
within  the  permissible  and  planned  ranges.'     To  accomplish  this, 
you  should:  ' 
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document  all  final  project  costs  as  deter- 
mined through  con€ract  negotiation,  and  * 
compare  them  to  your  original  estimates 
to  note  any  significant  differences 

display  all  of  the  line' item  costs  of-  the  program 
to  ensure  that  ♦funds  are  being  planned . to  genei^te 
services  at  maximum  levels  of  efficiency  and 
effectiveness  in  tife  areas  of:  *  ^ 


^  i  -  Admiai  strati  on 

r  -  Allowances 


-  Wages 

-  Fringe  Benefits 
<-  Training 

r-  (Services 


^  array'  costs  'within  the  jgrpper  categories  to' 

.  ,         ensure  that  all  ,pro  jedStl  co^ts  are  ir^^cord- 
/>  *  *     ance  with  financial  -provisions  of  the  Federal 

Regulations:  . 

Participant-Related  -  .at*  least  50% 
"       EGS  f  up  to  30%  ,  '     +\ • 

'Administrative  -  no  more  than  20% 

-  .      '  \ 

'  •      ensure ,  t^haA  your  total  cost,  does  not  exceed  ^ 

m       your  tbtaAjTitle  VII  grant  allocation,  unless  v,v 
*       other  available  resources ^(federal ,  state,  of 
local      public  or  private)  hav£  been  committed 
to  the  program.  -? 

-  <  '  -   \  '      -    \,  t 

If  your  ^pafysls  trevea*ls  any  problems  in  the  budget  you  „  ■ 

should  make  the  last  minute  adjustments  that  are  necessary  to 
bring  your  total -program  into  compliance.  -  -  . 

* 

Operations  Plan,  *  . 
After  all  individual  work  plans  are  developed  and  cost 
^estimates  have  t&en  checked  and^ finalized, • this  information 
|  ^houlfe  be  consolidated  into  a  master  operatipns  -plan  for  the 
k-FIC.     Tliris  master  operations  j^drktshould  incorporate  the  manage 
ment  and.  evaluation  procedures  axiA  checkpoints  from  your  .PIC1  s 
Management  and  Evaluation  plans  into  standard  bperating  proce- 
dures1 for  program  operations.     At  a  minimum,  your  PICfs  final 
operatipr^plan  should  dfeScribp:  .  * 

*  •      PIC  Prcigram  Performance  Standard 

«'     PIC  Administrative  Structure  and  Organization  ' 

*  ^  a  + 

•      PIC  Operating  Procedures  *  0  \ 
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PIC  Program  Delivery  System 

PIC  Projects  -  • 

PIC  Management  "^Lafri  and  Procedures 

-  PIC  Evaluation  Plan  and  Proceflures 

!-   B^ciget*  Allocation  and  a  Accounting  Procedures* 
*  • 

Once  that  plan  is  developed,  your  PIC  Council  should 
endorse  it  as  a  plan  ahd  prj^gram  approach  that  all  members 
can  accept  and  support.     *  ^  * 


Be  prepared  to  Re^lan  ^  \/  , 

A'^final  i^ote  on  you^  Program  Operation  Plan  is  that  $1- 

thouqh  considerable  time  and  effort  goes  into  creating  a  plan 

*  S  — 

which  is  as  appropriate  and  responsive  as  possible' to  local  ^ 
ne^Js,   the  plan  is  based  upon  fs&Ttors,  as<Wpt i^n* ,   and  infor-- 
mation  at  a  given  point  in  time.     The  world  is  not  static. 
Ch^ges  in  the  ecqpofny,  size  and  needs  of  target  groups,  and  ^ 
CETA  funding  levels  ,  >amoTig  other  things,  will  cause  the  need 
"for  replanningS  When  this  need  arisen  .your  PIC  should  do  its 
replanning  using '  the%ame  ^roce^that  you  employed  ^o  develop 
its  original  operations  plan. 
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E.  DEVELOP  MANAGEMENT  * (MONITORING)  PLAN 
i  J, 


E -PLANNING  FEEDBACK 


E.  Divilop 
Minigilnint 

Plan 


I 


C.  EftikilUi 
Prefrim 
Sctfi 


0.  Dmtif 
Prt|fiin 


I 


F.  Dtvtlfi 
Ewhttba 


G.  MtMtr 


ose : 


Steps : 


J 


To  establish  a  detailed  plan  for  overBfling 
your  Pica's  program  operations  on  an  omjoing 
ba  s  i^"^^  \J  y      «  *■ " 


1.  ;  Establish  Management  Review  and 

Contiol  Process        *  y 

2.  Determine  Man£gement^  Information 
and  Reporting  Needs  V  * 

3-     Design  Management  Reporting  System 


I* 

V 
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Outcome ; 


Use: 


A  written  descrioMon  which^clearly  delineates 
your  PIC's  apprdach  to  it^nagembnt  review  aiW 
analysis  of  program  operation^'  and  its  procedures 
,for  reporting  a  nd#  ut^l  izajiioa  of  ^management  t 
information.  .  .-^     *  V  *     *  4i 


'A 


The  Management  Plan  is  ystfdf  t 


o  cTiV 


s 


ect,  monitor, 


V  ^id  control  your  P'lC^s  program  activities.  \  It  ^ 
•t  \  enables  you  to  identify  .problematic  areas  and  to 
)   *  "take  correctfJfceJ^actions  as  necessary*.  % 
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1.     Establish  Management  Review 
*     "  And  Control  Process 

'   ,The  purpose  of  this  ?tep  is  to  establish  specific  respon- 
sibilities and  steps  for  management  review  of  your  PlC'-s  pro- 
grahv^   This  involves :  • 

identifying  your  PIC's  operational  control' 
process  and  the  basic  steps  required  to,  .  % 

monitor  and  control  program . performance 

dividing  responsibilities  ^fcrr ^management 
review  and  control  among  PIC  members  and 
/staff.  ^  p 


The  composition  of  the  Council  enables  you  to  gain  the  insight 
of  a  varidty  of  organizations,  part Lgularlyf business .    ^It  is 
important/ that  the  Council  play  a  mdpnln^ful  role  in  the  manage- 
ment decisions  so  that  its  unique  perspective  can  be  an  ongoing 
part  of  PIC  activities.         t  < 

Basic  Control  *  Procedures  ^  ^ 

In  order  to  determine  your  PIC's  basic  control  procedures, 
-you  should  ask  the  following  questions:  J 

^       %    ♦••What  information  is  absolutely  essential 
for  monitoring  perf ormance  and  managing 
^  the  PIC!s  program?  •  * 

% 

•  What  are  the  sources  of  this  information? 

•  How  can  this  information  be  collected?  .  •  ' 


•  What  procedures  ate  necessary,  for  analyzing 
the  information?  *  * 


•  When  and  in  what  manner  should  tjie  informa- 
tion be  reported?  * 

Who  will  review  management  reports?  0  %  ^ 


#*How  will  corrective  actions  be  implemented; 

Answering  these  questions  will  enable  you  to  establish 
the  basic  framework  for  your  PIC's  management  review  and  con- 
trol process.  '  It  will^allow  you  to^cide: 

"  *  •'  8t>. 


•  the  amount  and4  type  of  data  and  information  v 
th,at  you  nee*d  „  ^ 

•  whether  data  can  be  retrieved  through  exist- 
ing reporting  systems  or  whether  new  forms  1 
will  have  to  be  devised  *  .  #j 

•  how  often  you  want  to  obtain  data  for  manage- 
ment purposes 

ft  M 

•  the  exact  process  for  reviewing  management 

data.  - 

The  PIC  and  the  Prime  Sponsor  Jnave  overlapping  -  interests 
♦  *  \  T 

with  regard  tp  management  procedures,  ankl  yoii  should  discuss 

these  early  when  you  establish  yoi^r  framework. 

.  r  To  irt&ke  the  foregoing  decisions  youV^sLlJ^  n^ed  rto4  know 
the  extent,  and  quality  of  the  et4ta  collected  by  the  Prime 
Sponsor  and  how  that  data  is  analyzed.     The  Prime  Sponsor's 
regular  system  may  be  useful  to  you  and  you  may  want  to  share    .  * 
resources  and  relevant  data.     As. another  possibility ,  the  Prime 
Sponsor  mig*rfe- be  able  to  provide  assistance  to  you  in  structur-  ^ 
ing  your  (^nonitoring  systj^    The  exact  nature  of  your  relation- 
ship to  theN^rime  Sponsor  in  this  area  is^a  critical  one  to 
jtfhich  your  PIC' should  devote  considerable  attention. 

Management  Review  Responsibilities  t 
Af ter^QU  have  made  your  decisions  regarding  your  opera- 
tional review  process- and  the  relationship  to  the  Prime  Sponsor 
you  will  have  the  information -necessary  to  determine  basic 
management ^review  responsibilities  within  the  PIC.  Questions 
you  -will  ?need  to, answer  in /this  area  include: 

/  -  * 

•  •  Who  will  ha\^e  ultjlma^e  responsibility  for 

administering  the .modi toring  plan? 
p  "      •  Who  will  ..supervise  monitoring?  , 

•  Who  will  prepare  monitoring  and  management 
reports?  - 

.  <     •  Who  will  be  involved  in  the  analysis-  of 

monitoring  reports  and  An  the  interpretation 
1  gf  findings?  \ 

•  What,  will  the  Counci^>^fole  be  in  the 
, monitoring  process?. 
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-  The  last  question  is  particularly  important.     If  your 
Council  is  to  have  an  active  arfTmeaningf  ul  role  in  the  man- 
agement of  the  PI-C's  prog^ramfr/  it^must  be  involved  in  monitor-, 
ing  the* program  throughout  the»year  and  participate  in  decisiorfs 
concerning  changes  in  direc  on. 


2  .     Determirre  Management '  Information  v 
'^fid  Reporting  Needs 

.The  initial  step1 in  identifying  the' specific  manage- 
ment information  ant  reporting  needs  for  your  "PIC  is  to 
review  the  objective^,  plans ,   and  standards  for  each  project' 
to  determine  preciselV what  needs  to  be^measured  and  re-' 
ported,  on      regular  basis ,  j± or  management  and  monitoring 

purposes,  ^ 

riot  all  PIC  activities  can  or  should  be  monitored.*1 
The  most  efficient  method # for  defining  reporting  needs  is 
to  analyse  each  project  against  these  criteria: 

•  What  activities  must  be  measured  and 
reported  by  law?  ^ 

•  What  activities,   if  measured',  will  give 

you  reliable  information  on  how  the  pro-  r 
.    gram  is  progressing? 

•  What  activities  should,  be  monitored 
because  they  are 

-  new,  q 

-  ■  exper icnental ,  *  v— 

-  politically  sensitive,  or 

-  expensive  in  terms  of  resource  -     >  1  ^ 
investment? 

Based  upon  this  analysis,  you  can  determine  what  ydu  should 
monitor.  •  ^  -\  '  *  .' 


V 
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Specifying  Inforimtion  Needs  •  ^ 

^    Once  you  have  identified  the  areas  in  which  you  need  to  \ 
measure,  the;  next  step  is  to  define  for  each  objective  or 
^rea  to  be  measured: 

#  .indicators  of  per foi^ftance 

•  the  information  you  will  need  in 
order  to  monitor 

9 

9  \ 

t  •    .  the  source  of  .this  information  and 
.  w       -the  "f  requency ;  with  which  you  should 
v  .  .  »      collect  it  ' 


■k  checklist  on  information  availability 

t  ■ 


j>er£prmance  .Indicators  ~  \ 

^       Performance  ipdicators  are  factors  or  items  which 
:    your.  PI£  can*  mi  as  tire,  for  monitoring  purposes.     They  can  be  — 
^athei;  qua-rttita(£i|>e  dr  quali  in  nature,  and  apply-  to 

th6"  tlC'*  total -pfAgrwf,  to  components  of  the  program, 
'J^-g- f  classrepom  training)  ,  , or  td  particular  projects  | 
•  (e.g.,-  an  ,EGS  'marke^n\  aafeivij^J  . 
,  '     '  Example  lQtficatS^  have  ±>een  cit^d  earlipr  under  the 

"Establish  Prpgrrfm"  Performance  Standard's"  step .     To  repeat 
"last  a  fewr'  %    *  \ 

^General  Indicators  ;     enrollment  "levels/ 
expenditure  levels'^   levels  of  service 
to  target  group©;  number  of  businesses  ^ 
training  ^>r  employing  participants.  . ,  ? 

Participant  Ef f ficfciven^es^  Indicators:  <r 
'  positive  *  termination  rates , *  entered  employ- 
.  itf^pt  ;*ates,   costs-per  participant,  cost  per 
'positive  termination. 

/  •     y         EG3  Indicators:     response  rate  to  Ipca^  ( 
'  employe^  survey;  number  of  jobs  added  as  a,, 

result  <$f  economic  envelopment  program; 
number  Of  formal  or  ihformai  planning 
agreements.  4  *  f 

To  establish  your  PIC's  monitoring  system,  you  need 
bo'  determine  which- indicator^  afe  mCst  appropriate  and 
then  to  gather  dat^  regarding  performance  relative  to  them. 


V*: 
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Planned  vs^  Actual  Performance  ~ 

The  most  common  form  of  assessment  in  the  employment  and 
training,  setting  is  a  comparison  of  actual  against  planned 
performance  to  determine  th£  degree  to  which  standards  or  ob- 
jectives q^e  being  met.     While  in  the  past,  these  comparisons 
have  o^teh  been  limited  to  quantitative  assfssmer\*r  assessment 
Qf  qualitative  measures*  is  equally  important   (e.g.,  were  anti- 
cipated deadlines  met?) .   If  .you  have  selected  the  indicators 
that  you  feel  are  most  imp9rtant,  then, you  have  determined 
what  tx>  focus  on  for  measurement  purposes.     The  important  con- 
side^ation  in  this  step  is  to  select  only  those  measures  'and 
to  product  only  that  data  which  will  be  of ^immediate  use  in 
managing  an^  directing  your  PIC's  program  on  a  day-to-day  basis. 

J 

*  i  -  — 

'  3.     Design  Management  Reporting  System 

i 

The  last  step'  under  this  heading  is  tgf design  your  PIC's 
management  reporting  system.     The  purpose  bf  this  step  is  to  ^ 
ensure  a  system  which  collects  and  reports  only  that  informa- 
tion which 'is  netessary  for  management  purposes. 

Reportjmg  Requirements  .  c  ^ 

The  Department  of  Labor  ^s  defined  certain  reporting 
requirements  for  .yoiir  £IC.     At  this  point,  you  have  decided  y 
what  additional  data(will  be  needed.     Therefore,  your  .respon-' 
sil^ility  is  to  design  and/develop  those  additional  report  forms 
which  viiil  be  necessary  /for  your  internal  control.     In  develop- 
ing these  forms,  you  should  ensure  that  they: 

*      are  easy  to  complete     -  .  -  f  . 

m     •  f  gather  tlje  information  which  is  degired 

9      go  to  the  appropriate  people  for  completion. 


11-68      ,f)f)  ^ 


\ 


For  each  report  which  is_ developed    you  need  to  specify; 

•  purpose 

•  contents 

•  source  of  data 

•  frequency 

•  distribution. 


Management  Reporting  Display  and  Assessment  Formats 

The  data  produced  by  your  reporting  system  provides  the 
raw  data  fox  the  management  assessment  of  your  PIC.     In  order 
to  be  useful,  however,   that  information  has  to  be  organized 
into  meaningful  formats  and  analyzed  within  the  context  of  your 
PIC's  Annual  Plan.     Two  especially  useful  tools  which  your  PIC 
can  develop  to  permit  such  recording  and  analysis  are  manage- 
ment reporting  display  charts  and  management  assessment  reports, 

Management  Reporting  Display  Charts:,  Management 
reporting  display  ctiaTts"are  created  to  permit 
recording  and  analysis  of  data  at  the  program, 
component,  and  service  deliverer  or  contractor 
level.     They  prfesent  data  against  those  perfor- 
mance indicators  which  your  PIC  has  determined 
to  b£  most  important  through  its  process  of  pro- 
gram operations  planning.     They  can  be  developed 
to  allow  an  easy  review  of  trends  over  time  within 
one  arfea,  or  to  compare  performance  among  prpjects. 
They  also  serve  as  an  early  detection  mechanism, 
for  potential  problems. 

Management  Assessment  Reports:     Management  assess- 
ment reports  take  the  raw  information  on  your  dis- 
play charts  and  through  analysis  converts  .it  to  a 
formal  description  -on  tTie  status  of  your  PIC's 
program,  program  components,  and  projects.  Typical 
management  assessment  reports  include: 

-  Descriptive  Analysis  of  Overall  Program 
Performance  to  Date  * 

-  Component/Project  ^.evel  Analysis  of 
Performance  to  Date 

-  Descriptive  Analysis  'of  Participant 

Characteristics  » 
t 

+       -  Assessment,  of  Program  Responsiveness 
to  Identified  N*eds-„ 
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In  general,  the  most  usfeful  management  reports  provide 
the  following  types  of  information: 

•  A  description 'of  the  current  status  of  the 
component/area  being  reported  on  w 

•  Identification  of  perforrqance  that  has 
*    deviated  from  plan 

•  Analysis  of  the  current  situation  and 
analysis  of  the  cause  of  deviations 

•  Identification  of  those  ar^s  that 
require  corrective  action 

•  Recommended  actions.  , 

Usually,  Management  Assessment  Reports  are  prepared  quar- 
terly and  annually  t&  coincide  with  DOL  required  data.  However, 
special  circumstances  at  your  PIC  may  indicate  the  need  for. more 
frequent  reports  and/or  reports  that  provide  analyses  of  infor- 
mation in  addition  to  that  required  by  DOL.     As  in  the  other 
areas  of  planning,  management  affd  evaluation,  you  should  make 
the  determination  of  what  your  particular  local  needs &are  and 
"^then  design  your  repofting  system  to 'meet  those  needs.. 

/  '  .  ': 
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F.     DEVELQP  EVALUATION  PLAN 


HE-PUNNING  FEEDBACK 


.  i 


F.  Divtiop 
E  vi  tuition 
Plan 


E.  Davtltt 

C.  Monitor 

MiHi|innt 
Flan  - 

J 

* 

A.  Amu 
Ual 
Narti 

*r 

1.  Oaf*l»f 
PIC 

Stntify  1 

C.  EmblisJ) 
Prorata 
Sett* 

D.  Dtvi 
fr«| 

PlM 

ran 
at»i 

H.  Ewtoata 
Ra«ifxt 


Purpose  :,  •  To  establish  a  detailed  plan  and^  procedures  for  . 


Steps : 


v^luating  your  PIC's  program. 


Outcomes ; 

—  !»  


ERLC 


1.  Establish  Scope  and  Nature  o"£ 
Evaluation 

2.  Assign  Evaluation  Responsibilities 
and  Establish  Procedures 

3/    Construct  Evaluation  Research 
Design 

4.     Create  Research  and  Data  Gathering 


Instruments 


An  Evaluation  Plan  that  will  be: 

•  feasible  in  terms  of  manpower 
and  cost; 

•  desy^rTed  to  measure-  the,  most 
^J^n^icant  projects  and  pro- 

•  gram  areas;  « 
> 
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•  of  ^alue*  in  future  PIC  planning 
and^^ogram  pperations. 

A  sound,  real\stic  Evaluation  Plan  can  be 
used  by  your  PIC  to  document  the  success  or 
failure  of  its  programs  and  to  secure  reli^bl 
information  for  replanning  purposes. 

> 


1.     Establish, Scppe  and  Nature  of  'Evaluation  . 

t- *  t 

An  evaluation  is  summary  in  nature:     It  provide?  the  means, 
at  the  conclusion  of -an. entire  performance  period,  to  assess  the 
ef  fleet  .of  your  PIC's  programs  and-  activities .  .Your  PIC's  evalu- 
ation can  serve  a  number  of  purposes  including: 

•  identifying  problem's  and  deficiencies  in'  ^ 
y  current  operations 

•  •  estimating  the  effectiveness  of  your  pro- 

gram in  achieving  specified  results'.- 

•.     comparing  alternative  strategies  and 
tactics  : 

•  •    refininq  planning  assumptions  *  4 

•  measuring  gaps  between  .planned  outcomes' 
and  results, 

i 

* 

Most  importantlyf  evaluation  dat^  can  provide  feedback  on  your 

PIC's  program  planning ,  -  operations    .and  management.-     This  data 

can  be  used  to  improve  future 

.    •  '  \  .  .*>  •  ' 

•  programmatic  performance  < 

•  .organization  and  operations 

•  planning  and  management.    "  • 

Tq  make  yqur  evaluation  most  useful,   it^ should  be  planned 
at  the  same  time  that  your  PIC's  programs 'are  being  planned.  In 
fact,  your  PIC?1  S -program  operation,  plan,  objectives,  and  perfor- 
mance standards  represent"  the  primary  sources  for  determining 
precisely  what  to  evaluate.  .  The  purpose  of  this  strep  is  to 
review  that -plan  tp  identify  the  scope  and  nature  of  your  . 
evaluation. 

There  are  three  primary  types\pf  evaluation  wl\ich  your 
PIC  can -undertake : 

'Outcome:     assessment  of  pepf  prance  against* 
_  ,  y  standards  set  at  beginning  of  performance  ✓  _  K 

\       perTod  to  determine  sdjegree  of  achievement.  <t* 

Relative  Ef  f  ectxvenesjs  i  t  comparative  analysis 
of  projects,   tactics ,   service  delivered,  and/ 
or  program  components  to  asde-ttairi  differential 
<   '         *    benefits  .and  merits  of  each-. 

'     *  •  -*  .  .  *  • 

.  j.  ,  .*•'••' 

•     H773:         ,  ." 

» 


Impact:/  measurement^ /to  determine  the  extent 
to  which  the  PIC's  program  has  brought  about 
institutional  Change  -and/or  has  .accomplished 
its  program  goals.*  - 


Outcome  Eva  Ilia  tion      -  1 

— it*  ;   * 

\£t  a  minimum,  your  PIC  will  want  to  evaluate;  outcomes 

agai«rxst  t-hose  performance  standards  yhictv  you" set  in  developing 
your  program  operations  pl&iu     This  evaluation  can  be  done  very 
easily  by  dirfectly  contrasting  actual  versus  pifenn^d  perforfifyce 
.for  each  ar-ec^i^i  which  you  set  performance  standards, 

„.As  noted  earlier,   on  the,  participant  side  df  the  ledger, 
mucjj.of  the  quantitative  data  yqu  heed  to  pefmit  this  type  of 
assessment  at  the  program  level  can  be  retrieved  from  three  1 
reports'--  Program^ Status / Summary ,   Summary  of % Par ticipan^  Char- 
acteristics,  and  Financial  Status*  Summary       .  prepared  quarterly 
and  annually  for  the.  Department  of  Labor.  *  You  will  need  addi- 
tional* reports,  however  \   to  :coilect  participant  dfcta  at^  the 
component  or  sferv»ice  delivery  levei-     It  will  be  somewhat  more 
difficult*  to  obtain  data,  in' the'  E(SS.  areaV    However,  "it  }s  impor- 
tant  to  collebt  information; in  these 'areas ,  and  If  you  have  set 
performance  standards  then  'you.  can  design  a. -reporting  system  to 
collect  the  *necessary\data  to  tell  you  whether  the  standards 
wer*e«.*et .  *  ^  • 

*      An 'important  point  , to  remember  is>  that  not  all  of  your      ,  , 
Pit's  projects  or  s*rvic6s  can  or  should  be  evaluated.  Given 
limited  financial  ajid  labor  resources ,  you  should  restrict  Your 
evaluation  to  those  projects  and  program  components  v)hichv.,  in 
.yoytr  opinion.,  are  most --demanding  of  assessment.     In  general,  a> 
project  should  be  evaluated  if  it  >^s  any  of  the  following^  chajr- 

:  '  *    '    '  '  \  ' '   >  '\ 

it  is  new  % 

ife>  is  ongoing  and  h&s  never  been  assessed  -  > 

it 'is  in  an  area  4n  Vhich  a  large  investment 
of  rQs'ourcefi  and  manpower  has  been  made    m  * 

it.  is  experimental  in  intent  and  design. 


act-er  istiCvS 


r 
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*  Relative  Effectiveness  Evaluation  •  ■ 

Assessing  the  relative  effectiveness  of  the  component 

pieces  and  effects  of  yojjr  PIC's  program  is*  especially  useful 
*for  replanning  purposes.     Date^on^ relative  effectiveness  rcan 

help  your  PIC  decide  what  to  continue,  to  stop,  to  increase, 

and  to  decrease.    .Possible  areas  you  carT'ioQk  at  m^issessing 

relative  effectiveness  include: 

•  'similar  programs  (e.g.,  clerk  typist^  run* 
by  two  different  service  deliverers 

m      different  programs   (e.g.,  truck  driver"** 
1  ^nd  a.uto  body  repairman)-  using  similar 
tactics   (e.g.,  OJT) 

•  one  program  (e.g.,  direct  placement) '  in 
terms  of  effectiveness  in  serving  different 
client,  groups 

•  effectiveness  of  TJTC  in  increasing  employer 

willihgness  to  l^ire  GETA  eligibles  as  opposed 

to  OJT  or  other  incentives 

'j 

•  different  tactics*  (e.g.,  OJT  vs.  classroom 
training)   f  or  >  th^f  same  occtipation  , 

•  effect ivene§s>6£-  k  strategy  in  serving 
different  client,  groups 

•  value  of-  ftiarkerting  i-n  identifying  new 
jobs  vs.  value  of  economic;  development 

i  '  efforts. 

▼ 

Implementing  evaluations  of  the  relative  effectiveness 
type  would  require  more  sophistication,  additional  data  collec- 
tion) and  resources  on.  the  part  of  your  PI-C  than  strict  outcome 

or  result  evaluation.^ 
# 

Impact  Evaluation 

Impact  evaluation  would  require  the*toost  sophisticated 
methodology  and 'would  undoubtedly  be*most  difficult  for  .your 
PIC"  to  implement.     Participant  areas  your  PIC  might  want  to 
look  at  in  terms  of  impact  include: 

comparison 


Earnings  gain 

i 

Private  sector 
,  jobs  secured. 

Promotions 
•  ob^aiqed 


(PIC  participant  group  vs, 
group)  / 

(PIC  participant  group  vs, 
-  group) 

JPIC  OJT  participants  vs. 
#" 'comparison  group) 


comparison 
comparison 
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Other  posjfble  areas  which  could  be  fexamined  and  questions 
which  rtiight  be  asked  in  terms  of  your  PIC' s  impact  'include: 

Institutional  Impact  of  Title  VII  -  " 

•  Has  the  PIC  *  stimulated  new  business*  parti- 
cipation in  local*  employment  and  training  - 
efforts?  *  I    .  1, 
Has  the  PIC  stimulated  new  responsiveness/ 
on  'the  part  of  "traditional"  employment  J 
and  training  deliverers  to  the  iieeds  of 
business?  , 

•  Is  there  a  comprehensive  planning  approach  » 
between  PIC  and  the  local  Employment"  and 
Training  Council? 

•  <Has  the. PIC  stimulated  any  areawide  planning 
between  different  prime  sponsorship  areas  and        *  « 
other  PtCs? 

Attitudinal  Impact  of  Title  VII 

•  How. has  the  PIC  program  affected  the  atti- 
tudes" of  employers  toward  the*  program, 
toward  employment  and  training  programs 
in  general,  and  toward  eligible  clients 
as  .potential  employees? 

•'What  impact  has  the  PIC  program  had  on  tHe 
attitudes  and  motivations  of  participants? 

Patterns  of  'Service        *  S '  V 

•  What  are  the,  characteristics  of  the  parti- 
cipants being  served  as  compared  to  regular 
CETA  clients,  goals,  and; some  statement  of 
"universe*  of  need^^  % 

m  What  are  the  characteristics  of  the  -busi- 
— Besses  being  served?    Where  do  they  stafid 
in  the  overall  economic  structure  of  the 
area? 

Cost  ^  - 
 , —  , 

•  What  reasonable  cost-benefit  calculations 
can  be  made  about  some  of  the  above  impacts? 


The  Evaluation  Decision         *  (, 

.Your  PIC  should  'decide  what  form(6)  of  evaluation  to 
implement  based  upon  its  decision-making  needs.    As  suggested, 
you  should  at  least  do  fen  outcome  evaluation*  in  those  areas  which 
you  consider  to  be' of  high-priority  for  program  success  ancTin^ 
which  ypu'need  information  for  replanning  purposes.     In  addition, 
since  PICs  are  often  engaged  in  as  many  innovative  efforts 


,  as  possible/*  yah  should  supplement  this  basic  evaluation 
approach  with  relative  effectiveness  and/or, impact  evaluation 
^measures  so  that  \he  valhe  of  these  efforts  can  be  compared 
to-  more  traditional  approaches.  x  ^^-^ 


2f  1  Assign  ^valuation  Responsibilities  ' 
~~      and  Establish  Procedures  , 


Once,  ♦you  hatfe  decided  on  the  scope  and ''nature  of  your 
PIC's  evaluation,  the  next\step  is  to  i^htify  the  basic  tasks 
tQ  be  performed  .and  assign  responsibility  for  their  performance. 


This 


esttebli 
ay sterna 

.detterm 

•    /  sel'ecfcil 

•  L  •"••/' 

As  iJn.the/de 


r 


procedures  fo*  an "organized 
approach  \p  the  evaluation 

fh^role  of  the  Council  ' 

^f  to  admitiireter.  the  evaivjfetion. 

V  '  ' 

Dment  of  your  Vn&nagement   (monitoring)  planj 
a  basic  'cc^r>Side££tion  here  must  t>e  tlie  quality  and'  scope  of 
the  Prime,' Spondfor 1  s  evaluation  syst>em.\  If  ^hat  system  is 
adequate,  you^sljiould  probably  Interface  Vlth/it  and  rely  on 
it*  to  collect  and  array  tl^e  data,  and  generate  the  reports 
that  you*  need  /for  evaluation  purposes^    T^s%  vhslsl^educe  the 
'letfel  of  fs£aff  effort  that  you  have  to  put  into  thefevaluation 
prooess*  i  * 

If,  on  Ithe  other  hand^  yotr^fiKd  that  thk^  Prime^  Sponsor 1  s 
system  is  inadequate  for  your  PIC's  needs,  then  you  will  have 
to  expand  yqjur  responsibilities  accordingly.     Because  of  the  . 
uniqueness  of  many  of  the  PIC  activities,*  particularly  EGS, 
it  .is  provable  that  you  will  need  to  supplement  the  Prime 
Sponsor's  regular  evaluation  system, to  assess  these Vctivities . 


•::•'-/ 
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'Evaluation  Review  Process  '    1  .  - 

The  next, task  which  your  PlC  should  ^accomplish  within 
this  step  is  to  establish' the  basic  operating  procedures  and  m 
working  arrangements  for  administration  of  your  evaluation  - 
efforts.     These  items  should  be  specified  at  the  dfbtset  of 
your  evaluation  planning.    General  questions  £o  be  answered 
include: 

•  Who  will'havfe  ultimate  responsibility  for 
i     administering  the  evaluation  program? 

Who  will  4w£rvise  the  evaluatOr  (s)  ? 

•  fcfcat  prdeedures  will,  be  demised  for  the 
'systematic  review  of  evaluation  plans ^ 
methodologies,  data,  and  findings? 

^  m  Who  will  be  involved  in  the  analysis  of  v 
,     <    "     evaluation  reports  and-  in%  the  interpre-   ^         .  • 
tation  of  findings? 

*  •  What  steps  will  be  instituted^  ensure 

•  -    *   that  deadlin^  are  observed? 

Evaluation  Personnel 

f     In  order  ,to  determine  who  should  be  responsible  for  your 
PIC's  evaluation,  you  should ;ansv^r  the  following  questions: 

•  How  much  enphasis -will  be  placed'on 
evaluation  at  your  PIC? 

•  What  resources  will  be  made  .available 

•  to  implement  your^evaluatidn /plan? 

•  Do  any  Axrrent  stfcff  members  .or  Prime 

Sponsor  staff  have  the  capability  (or      ;  •  f 
the  time)  to  perform- evaluation  research? 

-#f  Would,  it  make  mo?re  'g^nse  to  hire  a  full-  * 

*      '    time  evaluation  specialist  or  gmploy 

private  contractors  on  an*  ad  hoc  basis?  .  \ 

•  Is  an  outside  evaluat^  necessary  to  .  . 
ensure  an  objective  *  perspective?  § 

-  •    .    »  .    .       •  • 

Council  Role 

.*    *  "  Regardless  of  how  'your  evaluation  is  done  *  by  staff, 
the  "Prime  £ponsor^  or  an  Outside  individual  -:#  it  is  important 
tha-tyour  Council  play  an- active^ W  in  the  administratis  of 
evaluation  activities.     Possible'  options -include:     .  . 

*         *   y  i 

•  i 

'  t 

.......     • .  '  •  •     -  ■ 
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•  appointing  a  Subcommittee  of'  the  Council  -  - 
to'  review  and  monitor  all  evaluation 
activities  ,q 

•  having, the  evaluator(s)  report  to  the 
Director, 'who  periodically  apprises  the 
Couacil  regarding  the  evaluation  program       •  *  4 

i  and  its  progress. 

Whatever  the  role  ^settled  upon  for  your  Pic)  you  should 
begin  planning  for  .your  evaluation  by  clarifying  roles  and*re-^ 
sp^nsibilities  and  .determining  the  basic  procedures  related  to 
the  administration  of  the  evaluation. 


c 

3.     Construct  Evaluation  Research  Design   *  ~ 

4        The  next  step  after  determining  the  scope  and  'nature  of 
your  evaluation  is  to  construct  your'piC's  evaluation  research 
design.     An  evaluation  research  design  is  a  formal  written 
statement- of  the  detailed  plan  for  implementing  your  PIC's 
evaluation.  1  , 

<^ 

Evaluation  Research  Questions 

^        To  prepare  yoar  EIC's  evaluation  research  desigA,  you 
should  ask  yourself  the  following  gu^stions: 

%       •  Why  is  the  PIC's  prografn  being  evaluated?^ 

•  Whfet  are  the  key  Objectives  the  PIC  hopes  to 
.  accomplish? 

•  What  are  the 'primary  and  secondary  objectives 
-(standards)  of  the  prpgram? 

•  Which  of  these  objectives  (standards)  are 
stated  in  readily  measurable  terms? 

•  Wftat  are ^the  activities  that  need  to  be 
well  understood  due  to  factors-  such  £S  the 
level  of  funding  they  require,  innoVativef. 
ness,  etc? 

» 

•  What  respurces  are  available  to  evaluate  the 
program? 
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^.  •  How  mucn  time  is  permitted  fo*  the  evaluation? 

•  What  data  and  other  information  is  needed?   *  1 

•  Where  does  this  data  exist  or  .whjgre  can  it  % 
be  requested?  •  ^ 

•  ^tow  can  this  information  be  obtained  (e.g., 
v  numdrtdal  reports,  narrative  reports  <£^T 

objectives,  surveys)?       ✓  •  ' 

•  How  will  this  information  be  analyzed? 

•  How  will  the<  findings  be'used? 

-  #  What  format  will  be  required  for  their 
presentation?  \ 

« 

The  answers  to  these  questions  will  structure  the  format  of 
your  .evaluation.     This  format  should  be  presented'  in  your  de 
sign  plan.  *  •  „ 

"     '  .   '  '  'J      ~  ^  \ 

Evaluation  Research  Design  Plan  |  £ 

Vour  PIC-1  s  formal   plan  for  its  evaluation  should  con- 
sist of  the  following  sections:        /  " 

Statement  of  Purpose':     states  the  reasons*  why 
the  study  is  being  undertaken..  Incorporates 
a  brief  description  6f  the  program  and  its 
goals.  '  / 

-Objectives :     specif iest objectives  and  scope  ' 
for  the 'evaluation". 

Methodology':     the  heart  of  the  research  design, 
states  exactly  how  the  evaluation  will  h&_ per- 
formed, f  Details: 

•  *~    *  • 

»  •  data  that  will  be  needed; 

techniques  of  data  collection;  and 
•  means  of  data  aggregation  and  analysis. 

Instruments:    describes  the  , types    of  instru- 
ments and  worksheets,  needed  to  collect  and 
treat  the  data  arid  idiscusses  the  process  by 
which  they  will  t>e  developed. 

,    .  <  Schedule:     sets  out  a  timetable  to  govern  the 
progress  of  the  evaluation.'   Highlights  all 
,  deadlines  and  due  da%tes. 

:  Reports:     indicates  the  types  of  reports  that  /  « 

will  resillt  from  the  evaluation,  inciudincf 
their  content,,  format,  and  submission  dates. 
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k  4.    Create'  Research  and  Data  Gathering  Vlnstfuments 

•  ~  T       •  ' 

The  purpose  of  this  st6p  is  to  develdplAll  researchr 
and- data  gathering  instruments  necessary  to  | implement  your 
PlC's  evaluation  plan* 


Baseline  Data  Worksheets  „  x 

If  your  PIC  restricts  its  evaluation  to  the  outcom^    •  m 
area,  most  of  the  data  you  need  will  be  easily  retrievable, 
either  through  required  reports  generated  for  the- Depart- 
ment of  Labour  or  your  PIC'1  s  internal  management  reporting 
system.     Therefore,  the  primary  data  gathering-  instruments 
you  will  have  to.  create  are  baseline  data  worksheets.  *  These 
worksheets  are  used  to  summarize  performance  .against  plan 
or  standards.     A  simplified  version  Sf  a  baseline  data  work- 
sheet is  presented  b^low. 


Project/ 
*  Area 

Indicator  . 

Standard 
or  Plan 

Actual 
Performance 

%  of  Variance 

 r  = — 

• 

Other  Data  Gathering  Tools 

If,  on  the  other  hand,  ypur  PIC  bro.adtfhs  the.  sqope  of 
its  evaluation,  there  are  two'  comiflonly  used  instruments; 
-you  might- choose  to  ^mploy:  , 

Survey  Questionnaire*-  an  instrument'  de-  * 
„  signed  to  .capture'  opinions  or  attitudes 

regarding  aspects  of  your  PIC's  program. 
The  questionnaire  may  consist  -of  forced-  ** 
choice  questions;  scaled  it^ms;  or  some  - 

other  form  of  quantifiable  responses. 

*  -  *>  *  * 

Interview  Guide  -  a  format  which  provides1 

a  sequenced  and  comprehensive  list  of  open- 

^     v  ended  questions  to  ask  in  a  personal  int^lr-  ■  J 

viiew.  . 
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Developing  Research  Instruments  -  *  , 

*  These  instruments  should  be  designed  to"  capture  as 

much  data  as  poss'ible  iji  the  simplest  and  most  efficient 

manner.  In  constructing  them,  yo^should  take  the  following 
factors  into  account:  '  — 

*  •    '    restrict  tfre  data  collection  effort  to  ^ 

*  that  information  required  to  achieve 
the  objectives  of  the  evaluation 

•  ensure  that  your  instrument  is  compre- 
hensive enough  In* scope  tp  answer  all 
questions  "which  are  important  to  your  f- 
evaluation  ief fort  <  * 

•  ,  make  the  wording  of  items  %imple  and 

nontechnical  so  that  they  can  be  easily  '  ' 

interpreted  by  respondents 

if  technical  terms  must  be  used,  define  % 
them  so  that  they  can  be  understood 
uniformly  by  both  the  respondent  and 
s  the  data  collector 

]  m       ^design  the  instruments  so  .that  the^  . 

"are  streamlined  in  format  and  e^sy  to  , 
use.  4*  . 

'  In  addition,  *the  instruments  should  be  pretested  prior 

to  their  use  in  the  evaluation.     This. will: 

«  » 
»        ensure  a  uniform  urjderstandinq  of 
-m   instructions,   items,  and  response 
categories? 

•  w     'confirm  whether  or  not  the  desired 

information  can  be, acquired; 

•  provide  an  index  of  the  timfe  required. 
f<3^  the  instrument's  'use; 

•  provide  a  perspective  01^  the  advis- 
ability of  including  any  "sensitive" 

;  or  controversial  items;  and  . 

•  provide  an  informed  opinion  regarding 
fche  format  anc^  appearance  of  the  instru- 
ment as  well  as  of  the  sequencing  of  the 

«'  _  items  on  the  questionnaire.  •  . 
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I      Adhering  to7 these  criteria  will. greatly  enhance  the  / 
utility  of  any  special  fev^Uuation  research  instruments 
th*at*  your .  EIC  has  to  create .  • 
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G.     MONITOR  PROGRAM 


•K  Amm 
Ual 
Umii 


BE-HAHHIRG  FftOIACK 


I.  Offtot 

rtc 


EmbBsJi 

Prtff«« 


0.  Dtvt 

v. 

Dptratfca 

Mm 

as  nec^Jsary. 


Purpose;    To  assess  and  redirect  program  activities 
I 


Steps; 


1.     Analyze  Planned  versus  Actual  1 
Performance 


j>are  Corrective  Action  Plans' 
^  .  Monitor  COfre 


v  }\  Prer 


Outcomes : 


.  Monitor  COTrectiye  Actions 


Monitoring  reports  which: 

m  document  the  PIC's  progress/, 
formance  according  to  plan 


•  ide 


«  Corrective 
' in  order 


derdand 


if y  problem  areas  which 
corrective  .action. 


action  plans  'to  be  implemented 
to  solve  operational  problems. 
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Your  £lC's  program  monitprincf  plan  is  its  4 
'"early  warning,H  systeftu ''  It  allows  you  to 
assess  .program  performance  on  aii  ongoing  * 
basis;  and  to  change  courses  or  methods  itl 
response  to,  problems.     In  addition,  it 
provides  information  for  program  evaluation, 


\ 
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1.  Alia] 


alyze  Planned  versus  Actual  Jerforpianqe 

\  J  ■     .  t   -  •        ;         .  . 

As  discussed  ^'arl^err  ,the  primary  means>.£6  monitor  your 
PIC's  performance  is  by  measuring  the  re!af€iOnship.  between 
planned  and  actual  performance.     The  purpose  of  this 'monitor- 
ing  is  to  assess  program  and  project  activities  on  an  ongoing 

bas.is.so  that:  ^  *  ' 

»,  *  • 

*       #     ,  problems , may  be* detected* as  3oon  as  they 

emerge^  >  {  '  / 

#v      sources  of -^problems  may  be  isolated 

»       remedial  action  may  be  .instituted  while 

the  program  or  project  is  still  in  progress*.* 

Your  PIC's  kanaq&nent  reporting  system  will  provide  $he 
basic  information  needed  to  review  performance  against  plan. 
By  arraying  this  information  on  your  management  Reporting  dis- 
play ch»fes,you  will.be  able- to' identify  ar^as  in  which  there 

•  ,  * 

ma^  be  problems.  , 

To  facilitate  your  review  of  this  information,  you  should 
establish  in  advance*  a  cqn'fcjrol  limit-     A  control  limit  is'  a 
prespecified  rtmge  of  tolerable  variation  from  pla^nne^l  levels 
or  scheduled  activities  yithin  which  signif icartt%  follow-up* 
monitoring  or  ccjrreetive  action  efforts  wi^l  not  be  taken;. 
Control  limits  enable  your  PIC  to  restrict  its-  corrective 
monitoring  to  those  areas  which  ar,e  critical. 


r  2.     Initiate  Corrective  Monitoring , 

 "  r  J  

*  i  • 

-  Ybur  analysis  of  performance  against  plan  will  enable  you 
to  pinpoint  areas  in  which'  there  m&y  be- a  problem.  t  However, 
your  review  of  report  data  will  not. indicate  what  the  source,  of 
that  problem  Is.     Therefore, ^if  you 'discover  a  significant  devi- 
ation from  p£9na,  you  should  initiate  ^a  more  in-depth  rnquiry. 
Appropriate  techniques  %include:  j 
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\,  #    .    on-site  monitoring  -*  when  data  must  be 

collected  from  participant ^or  contractor  . 
files,  or  when  raotp  qualitative  infor— 
matioivis  desired  •  ,  * 

"  -  :  •  i  r  . 

-    •      ,  interviews  -  with,  participants^,  employar^, 
or  contractor  staff  to  collect* data  which 
carijiot'  be  obtained  .f rota  standard  source?- 

By  doing  this  foU.ow-up  in  problem* areas,  you  can  determine 
"whether  corrective  action  is  necessary  and  secure  the  es 
tial  inf6rmation  >for  .preparing  a  corrective  action  plan. 

To  ensure  that  your  PIC  dofes  4n  effective  job  of  /follow 
up  or  corrective  monitoring ;*you  should  establish  a*  cfefunite 
procedure  for  doing  corrective  monitoring.    'Some  Aints  on 
implementing  an  effective  collective  monitoring  process 

iaclude :     fi  t  ■ 

*»  # 

#  manage  by  exception:     concentrate  on  thos^^ 

areaS. which  appear  most .problematic  "*W 

j?       *  *  *  ■  * 

-  0*       be  systematic:     in  advance  of  your  vi^it  ^ 
develop  all  questions  *hat  you  want  to  ask 

«       talk  to  the  project  operator  first:  before^ 
.  initiating  a  visit  contact  the  project  operr  1 

^ator  to  find  out  if  there  might  be  a  simple 
V  "explamttion  for  the  performance  dfeviation 

#  pursue  a'  "no  surprise''  policy :    outline  the 

•  stegs  that  you  intend  to  fallow,  communicate 
them  to  the  project  operator ,  adhere  to  them, 
and,  maintain  communication  with  the  project 
operator  throughout  the  review  process  *, 

•        •  •     %  -  v 

*  avoid  unnecessary  disruptions  to  project 
operations?     try  to  schedule  your  visit (s) 

*     so  that 'they  "fit"  best  With -the  formal * 

•  routine  and  operatibns  of  the  project- 

*  remember  -  the  plan  could  be  wrong:     reassess  • 
the  validity  of  your  planning  assumptions,  and  „ 

9  *    \  standards  as  an  integral  part  of  6very  correc- 

tive monitoring  visit..  % 
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Nw  3#    Ensure'  Necessary  Corrective  Action  *   ■      -  * 

\&  .  *    V  ' 

+ .    *  \  There  are  two  essential  ingredients  to  ensuring  necessary 

^  \ 

1       corrective  actions  in  those  areas  in  Which  significant  deficien 
cies  are  found  ^hrough  corrective  .monitoring.  •  They^are  '(1) 
develop  corrective  action  plans,'  and  (2)  irtonitor  tne  implemen- 
tation  of  those  plans,  /A 
^   *  N   Your  corrective  action  plans  should  be  deve/loped*  to 
"address  directly  the  specific  deficiencies  in  each  area  of 
program  operation  isolated  through  performance  ^oalyais.  These 
plans  should  specify  the:     %  /• 

•  •  /  .  , 

•  ...problem  and  its  source  ,  • 

•  extent  «of  its  deviation  from  pl^n  ,  ' 
.«        criteria  required* for  improvement    •  f 

rentedial  steps  to  lie-  taken  ,  /  \ 


-  ..time  frame  for*  improvement  .  \ 
-^assignment,  of  responsibility  for  action, 


You  should  normally  develop  corrective  action,  plans 
following  the  quarterly review  and 'analysis  pf  program 
performance^  While  performance  data,  are  generally  'Moni- 
tored and" analysed  on  a  monthly  basis;  it  is  more  useBuL 
to  withhold  corrective  action  plans  until*  more  time  has  ' 
elapsed*.     Preparing  corrective  action  pljjns  on  a  quart/ 


basis i 


dil^|^th^  influence  of  any  isolated 
mbn|HRiber  rat  ions  «      .  '$< 


m        provides  a  more.  comprehensiVe\and 
• '    representative  data  base  from  wfcich 
to  initiate  action  / 

*  allows  yovdta^take  a  more  thoughtful,  ^- 
analytical  approach  to  problem  remedia- 
tion. ^ 

Corrective  action  plans  ^should  be  negotiated  with"  and  ^/ 
approved'  by  PIC  management  and  appropriate  contractor  or 
project  staff..    The  chief  criterion -that "you  should  employ 
in  developing 'those  plans  Afhat  they  Remain  consistent 
with  overall  PIC  program  gols,  objectives,  and  standards. 
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Corrective  action  plans  must  be  carried  out  effectively 

and-  on  schedule  'if  they  are,  tc^edirect  and  improve  program 

operations.    Therefore,  you  should  establish  a  special 

monitoring  system  to  provide  you  with  feedback  on  the  status 
of  the  corrective  actioji  plan- implementation.    Such  a  syfetem 

involves  specifying  the:  1  fc 

"  tyPe  of  inf9rmation  needed 
•  -  schedule  for  implementation 
•        responsibilities  for  action.  ^  • 

<.  *  • 

^The  format  is  set  up  so  that  follpw-up  results  can  be 
noted  month,  by  month  within^ the  quarter,  until  all  three 
months'  activities  are  noted,  providing  some  indication 
of  progress .during  the  quarter.  ^ 

Your  PIC's  ability  to  monitor  program  and, project 
progress  and  to  redirect  efforts  as  necessary,  will  con-  * 
tribute-  significantly  to  the  ultimate  success  of  your  PSIP. 


( 
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1^     EVALUATE  RESULTS 

M+UMNM  MtOI^CK  - 


re? 


i.  Dtvftof 


F.  DmlM 
PUS 


6.  MiiHtr 
Pr«|fiM 


A.  A«mi 
Ual 
Umii 

1,  Drfit' 

PIC  t  • 

•  * 

C.  If&kB* 

0.  Dm 
ftf* 

nan 

••P 
Kit! 

» 

.Purpose:  Jfo  .asses 
^program: 


s  the  effects  and  outcomes  of  %he  PIC 


Steps; 


Outcomes 


o 

ERIC 


&  Collect  Data 
^2.     Analyze  Data 

'3.     Feedback  Into  Planning  Process 


E^aluat^ion  findings  which: 

#  document  the  PIC's  performance 
against  locally  established 
standards 

•  identify . factors  which  con-' 
tribute  to  ot  inhibit  the  success 
of  the'  program  , 

•'substantiate  the  cost-effectiveness 
o'f  ?the  program  and  its  components 

•result  in  a  clarification  and  • 
reexamination  of  program  'objec- 
tives  and  assumptions 
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m'm  isolate  the  need  for  and  naturfe 
of  any  corrective  action  necessary 
1  to  enhance  program  performance. 


The  evaluation  findings  are  the  primary  input 
for /planning  your  PIC's  program  for  the  following 
year.    Your  PIC  should  use  its  evaluation  results 
t<^  reassess  planning  assumptions  and  program 
assign  principles,  to  determine  project  and 
service  deliverer  effectiveness,  to  pinpoint     ,  • 
areas  to  change,*  and  to  begin  to  develop  an 
evaluation  plan  for  next  year* 


J 
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1.    Collect  Data' 

The  purpose  of  this  step  is  to  collect  all  data  which* 
is  necessary  for  evaluation  of  your  PIC1  s  program'. -   Y9ur  < 
evaluation  research  design  will  specify  the  data  gathering 
methodologies.     The  most  common  data  collection  techniques 
are  discussed  briefly  below. 

Baseline  Data  Collection 

'   *    Your  PIC's  internal  management  reporting  systeA  and*' 
reports  prepared  for  the  Department  of  Labor  can  be  used 
to  secure  baseline  data  on: 


•  c  participant  characteristics  ~ 

•  types  of  services  provided 

•  placement  data  4 
0       expenditure  information 

•  applicant  records 

•  participant  tracking 

•  product  development  and  activity 

accomplishment 

•  observations  and  monitoring  reports. 

These  baseline  data  should  be  collected  on  an  ongoing  basis 
for  projects  that  are  to  be  evaluated.     In  this  way,  they*: 

•  can  provide  leading  indicators  in 
*  advance  of  the  summary  evaluation 

undertaken  at  the  conclusion  of  the 
program 

'  •       will  not  become' a  time-consuming  task 
for  the  evaluator(s)  once  the  formal 
evaluation  gets  urtderway. 


Survey  Research  V  •  S 

Survey  research  is  a  reliable  method  of  g^ierating  data 
pertaining' to  respondent  attitudes  and  opinions  regarding  your 
PIC's  performance.     It  can  be  a  useful  means  for  evaluating 
both -participant  and  EGS  activities.     Surv6ys  can  ascertain 
respondent  views  on  any  subject  with  which  they  can  reasonably 
be  expected  to  be  familiar. 
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The  first  consideration  fidr  your  PIC  in*,  conducting 
survey  research  is  designating  the  sample  to  be  surveyed. 
When  the- sample  has  been  selected  and  the  questionnaire 
created,  tested,  and  refirifcd,  your  survey  is  ready  for 
implementation.    A  .suryey  may  be  administered  in.beveral 
ways:  m  9  1  / 

•  by  nmi  l 

•  yf  telephone  _ 
%       in  person 

•  individually  or  in  a  group. 

A  survey  may  also  be  either  anonymous  or  identifiable. 

How  you  conduct  a  survey  will  be  dictated 'by  two 
primary  concerns: 

•  the  extent  e*f  available  resources,  and 

•  the  need  for  a  high  response  or  rate 
of  questionnaire  completion-.  * 

In  nvosft  surveys,  there  will  not  be  a  100%  response 
rate.    Anticipating  that  some  individuals  will  not  return 
a  completed  questionnaire,  you  must  "over sample"  to  obtain 
the  required  number  of  Responses. 

Individual  Interviews  ♦ 

Individual  interviews  are  another  means. of  obtaining 
evaluative'  fcata  regarding  PIC  performance.^  These  interviews 
could  be  espedialLy  useful  in  securing  information  on  the  EGS 
facets  of  your  program.     They  provide  the  opportunity  for 
amplification  of  the  respondent's  views.    A  few  guidelines 
your  PIC  should  follow  to  enhance  the  prospect  of  a  success- 
ful personal  interview  inttLude: 

•  Interviews  should  be  scheduled  at  the 

sit^e  where  the  respondent  feels  com-  ; 
fortable         -  / 

No  interview  should  require  mord  than 
>  one  hour 

•  The  interviewer^  should  riave  full  recall 
?     /-          .of  the  contents' of  the  interview  guide 

and  be  as  familiar  as  possible  with 

the  subject  matter       *  \ 
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/The  interviewer  should  ask  follow-up 
questions  that  may  be  triggered  by.  a 
particular  artswer 

the  interviewer  should  seek  confirma-  \  f* 

fcion  at  frequent  intervals  during  the 
interview  by  restating  his  or  her  « 
understanding  of*  particular  responses  «/ 

***     *       *  *  *  * 

The  interviewer  should  prepare  a  *  « 

formal  narrative  based  dh  his -or  her 
notes  soon  after  the  interview  has 
been  completed. 


Group  Interviews  fS  , 

As  an  alternative  to  a  series  of ' individual  interviews,  ' 
a  >group  interview  is  an  inexpensive  and  effective  method 
which  your  PIC  could  employ  to  collect  a,  substantial  amount 
o'f  data.    The  basic  requirements  for  a  group  interview  are: 

•  the  prospective  respondents  must,  be  roughly.        \  • 
equivalent  in  terms  of  their  knowledge  of, 
the  subject  matter 

*  •  the  respondents  must  b£  willing  to  partici- 

pate openly  in  a  group  interview. 

With  regard  to  the  first  requirement,  the'  respondents  * 
ittust  be  individuals  wfyo,  for  example,  participated  in  the  same 
training  program  or  who  were  placed  in  similar  -jobs.    For  pur- 
poses of  the  evaluation,  in  other  words,  they  must  share  the 
same  principal  characteristics.    The  second  requirement  refers 
to'  the  fact  that  the  participants  must  understand  the  .nature 
pf  the  group  interview  setting  and  be  ready  to  express  the^r 
opinions  candidly  in  the  presence  of  others. 

A  group  interview  proceeds  in-tihe  following  manner: 
*  *  * 

•  the- assembled  participants  receive 
an  ori^entaC^H  to  the  process 


interview  guides  are  distributed  to 
all  participants 

the  guides  'are  completed  independently 
by  the  respondents 
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each  individual  then  t epojrts  his 
or  her  perceptions,  as  noted  on  the 
,  interview  guide,  to"  the  ehtire  group 

I  * 

•  all  responses  are^ recorded  on  newsprint 
by  the  facilitator  . 

•  there  is  then  a  group  c^idcussion  to 
\  clarify,  combine,  ana 'supplement  the 

data.  *  ,  >.  f 

The  product  of  .this  procesav  is  a  large  body  of  consensual 
data  for  every  iteph  on  the^  ii?tter»vi4w  guide. 


Follow-Up  Studies 

*    Follow-up  studies  are  a  specialized  version  of  a  survey 
or '  interview.     They  are  highly,  usepul  means  of  assessing  a 
project's  o^  your  PIC's  impact  aftier  a  participant  has  completed 

nded.     Such  studies  can 
determine  whether  or  not  a  prograA  had  a  continuing  effect  pn 
the  participants  or  built  desired! linkages  with  employers  or 
other  organizations.     *  s 

Follow-up  studies  are  best  /conducted  using  some  form  of 
survey       either  by  mail  or  tfelejhone.     Preparation  for  a  „ 
,follow-up  study  includes: 


#  "    determining  who  is  td  be  interviewed. or 
surveyed  (e.g.,  participants,  employers, 
'  etc.) 

I     '  V 

0       identifying  the  information  needed        ^  / 

m       developing  and  pretpsti-ng  questionnaire*' 
or  interview  guidegf  c  \ 

*  #       selecting  a  samplej 
The  data  are  then  collected, lanalyzed,  and  'reported. 
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^     2.    Analyze  Data 

Data  analysis,  or- data  treatment  as  it  is 'sometimes 
called,  is  the  art  of  viewing  data  f  rom  s  a  variety  of  per- 
spectives in  Grder  to  understand. and  interpret  it.  Data 
analysis  involves :  * 

•  reviewing  the  research  findings  for- 
correlations,  trends,  commona litres, 
and  disparities 

•  documenting  the  relationship  between 
the  findings  and  the  evaluation  objec- 
tives 

•  developing  rationales  or  speculations 
for  findings  whose  explanations  are 

riot  self-evidfent.  ^ 

There  are  four  basic  types  of  analysis  that  may 
useful  in  evaluating  your  Pie's  programs: 

Cost-effectiveness :    quantitative  index 
)  of  performance  that  relates  various  pro- 

gram results  to  cost. 

.Cause  and  Effect  Analysis:  .weighs  all 
variables  to  demonstrate  a  causal  rej^ftion- 
ship  between  two  or  more  factors.  _>  * 

■   ,  *  •    '         %  ' 

Comparative  Analysis:     focuses  on  relative* 

*  achievements  of  two  or  more. groups,  units. 

Trend  Analysis:    documents  changes  in 

levels  of  achievement,  over  time.  w  - 

As  these  types  of  analyses  indicate,  data  analysis  is 
much  more  than  a  mere  review  of  numbers.    It  is,  rather,  a 
systematic  attempt  to  derive  meaning  from  and  make  informed 
judgments  about  the  nature  of  the  program  b^ing  researched. 
Regardless  of  the  form  of  analysis  your  PIC  is  employing, 
you  should  attempt  to  answer  the  ba$ic  questions: 

\    #  Were  the  PIC's  or  the  project's  objec- 
tives achieved?  - 


11-97 


118 


m  What  caused  the  effe<5^3  discovered  in  the  - 
evaluation?  * 

•  Should  the  components ,  progf'ams,  or  activi- 
ties be  continued,  "changed  or  terminated? 


-3.    .Feedback  Into  Planning  ProCesg^ 

Onfce  your  PIC  has  analyzed  its  evaluation  data, the . last 
tasks  in  the  evaluation  process  are:  ^ 

•  'f  to  prepare  an  evaluation  report  which 

presents^  the  results  of  the  analysis 

•  to  ytilize  that  report  to  log  in  the  ^ 
review  of  program  performance  and  to 

plan  for  the  upcoming  year.  y 

*  •  ^ 

Evaluation  Report  c 

Your  PIC  should  present'  its  evaluation  findings  in  an 
evaluation  report.     The  degree"  of -formality  and  sophistication 
of  that  report  is  obvioVsly  a  matter  of  local  preference.  ^How- 
ever, since  the  evaluation  report  is  the  of ficial , record  of  the 
evaluation,  making  public  your  activities^ ^nd  findings,  it  should 
be  carefully  prepared.     The  basic  sections  which  should  be  incor 
porated  in  the  report,  regardless  of  if£  length  or  fopiat  are: 
Introduction  -  'to  the  program  *and  the  evaluation  ^ 
Objectives  -  of  the  evaluation  -  ' 

Methodology  -  desighed  to  conduct  the  evaluation 

Findings  -  based  on  the  data  collection  3nd  s 
*  analysis 

Recommendations/Conclusions  -  based  on  the 
findings  \  ~  ~ 

Executive  Summary  -  briefly  explains  the 
objectives  of  the  evaluation  and  high- 
lights the  key  findings.  ♦ 

4  * 

Assessment  and  R6plannjng  „ 

Your  PIC  should ;use  its  evaluation  jreport  and  findings  to: 

•  reassess  planning  assumptions  and  program 
design  principles  f 
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■  'J  •  *  '  ;  •  '   •  - 

.~  determine  prpject  and  service  delivered      '  « 

effectiveness 

•       pinpoint  ar^as  for;  change.  \ 

•  «  -»  .  * 

*  •     'begin  to  develop  its  evaluation  plan  for 

next  year*.  * 

The  evaluation  results  will  aid  in  determining  if  a  parti- 
cular project  should  be  continued,  changed ^or  terminated. 
.Evaluation  .of  -a  number  o^  projects  will  permit  comparisons  of 
each  to  determine  which  activities  are  most  effective  and  cost- 
efficient.    The  results  will  also  provide  a  perspective  from 
which  to  assess  the  appropriateness  of  the  PIC 1 s . performance 
standards.     In  summar^-r  your > PIC  should  employ  its  evaluation, 
findings  judiciously  to  "develop  a^  improved  PIC ;program  opera-, 
tions-  plan  for  next  year. 


V 


\  •••■■/ 


v 
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YOUR^PjC'S  RME  PROCESS  -  SELF- ASSESSMENT 


Before  proceeding  to  the  next  Section  on  how 
to  apply  the  PME  Process,  it  may  be  useful  tQ  once 
again  reassess  your  PIC's  present  PME  approach,  -^o 
do  this,  complete  the  detailed  checklist  which  follows. 

Use  your  ratings  of  individual^  .items  to  focus 
your  reading  of  Section 


7 
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*    Private  Industry  Xduficil  ' 

J  PLANNING/  l&NAGEMENT  AND  EVALUATION  APPROACH 

SELF-ASSESSMENT  * 

Please  reafpond  ta  the  following  questions  candidly.  Circle 
the  nurabet  which  £est  reflects  your  opinion*  Regarding  each 
itefln.  '        '  \  *  ' 


1.  Are  the  purposes,  goals,,  and 

role  of  the  PIC  clearly  definfed? 


'  2 


'  Not 
clear 


Very 
clea* 


Is  the  division  of  PME  roles, 
and  responsibilities  among, 
^PIC  staff  and  members  clear? 


Not 
clear 


Very.> 
clear1 


3.  Are  the  PME  roles  of  the  PIC 
and  Prime  Sponsor  clear? 


Not 
clear 


Very 
cleat 


4.  Does  the  PIC  have  a  well- 
defined  process  for  developing 
its  annual  service  plan? 


2, 


Not 
Well^  . 
Defined 


Weil- 
Defined 


5.  Doss  the  PIC  hcfve  a  systematic 
'way  of  obtaining  local;  labor 
market  information? 


*{°t 
systematic 


3      4    -  5 

v  Systema t £c 


6.  Does  the  PIC  have  a  stan-*  - 
dardized  or  objective  method 
for  reaching  decisions  on 
what  activities  to  undertake? 


Not 
stan- 
dardized 


4  ^5- 


Stan- 
dardized 


Does  the  PIC  establish 
measureabfce  objectives 
and  standards  for  all 
acfciyitids -which  it 
undertakes? 


Never 


Always 
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8.  Does  the  PIC  devfelo£  detailed 
work  or  jjfrpl emefrta tion  plans 
for  all  activitie5TTr»4  facets 
of  operation?  ^ 


Never 


/ 

4,     5  - 


Always 


9.  Does  the  PIC  have  a  systematic 
way  of  collecting  data  for 

"   program  management  and  moni- 
toring- purposes?  • 


10.  Is  the  management  system  de- 
signed to  include  indicators 
which  can  be  used  to  identify 
areas  of  performance  short-*  « 
,    fall  fpr  all  activities? 


Not 
system- 
atic * 


Not 

at 

all 


3      4  5 


4- 


System- 
atic 


4  5 


»  Com- 
pletely 


,11.  Are  all  management   reports  '  • 
*  *   easy  to  read  and  generated  * 
cqj  A  regular  basis? 

12.  Dogs  *the  PIC  tegular ly  monitor 
.and  review  program  performance? 

13.  Are  rftfciagement  reports  used  to 
take  corrective  action  and  to 
change  program  direction  when 
necessary? 


J«o  .  * 
1    '  2 


Never 


3      4  5 


Yes. 
/  * 
3-4  5 


Yes 


3      4  5 


Always 
* 


14 


15 


.  Does. the  PIC  have  a  systematic 
and  valid  plan' zor  evaluating 
the  eftjftc^veness  of  each 
activil^pTnd.  its.  total  pro- 

Does  the 'VtC have^a  way  of 
differentiating  the  relative  1 
^effectiveness  of  its  service 

deliverers? 

-  • .         -v       .  -  / 

lie        jise  ^er f  o  nuance  < 
^results  in  its  ' 
fotts? 


Not 

Com- 

at 

pletely 

all 

* 

1 

2  •  3 

4  5 

NO 

Yes 

1 

2  3 

0 

' .  4      5  ■ 

Never 


Always 
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XII.     APPLYING  THE  PME  P'ROCESS 

•   •  •:  f 

This  section  if  the  Guide  describes  "how"  you  can 
implement  the  steps  in  the' PME  Process.     The  steps  are  organized 
according  to  the  "flow  df  the  PME  Process"  and  presented  by  the 
stages  outlined  in  the  preceding  section. 


nCEMt  PROCESS  -  FLOW  CHART 


REf  lAttNIRG  FECOlACg 


C.  EmfcUtJi 
Sum 


Effbiifc* 
ftta 

G.  UlMM 

PME  Stages  / 

»  /  . 

Each  stage  is  introduced,  as  shown  on  the  following  page, 

by  a  brief  statement  of  the  purpose  for  the  stage  and  a  listing 
o'f  the  steps  within  the  stage. 
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STAGE  A:     ASSESS  LOCAL  NEEDS 


 '  \  # 

Purpose:  To  determine  the  current  employment  and 
  training  needs,  within  your  community. 


Steps 


V 


1\    Analyze  Local  Labor  Market  Conditions 
and  Business  Needs 

2.    Analyze  Local  Population  Groups  and 
Service  Needs  ^ 

3      Review  Programs  and  Plans  of  Other 

^gencies/Service  Deliverers/Businesses 


i 

PME  Steps 


Then,  for  each  s 
* 


as  follows: 

*  1. 


Listed r  in 


formation  is,  presented 


3. 


Introductory  page  which  explains  step , 
defines  "key  concepts,"  and  allows  you 
tojassess  your  PIC's  present  knowledge 
anGjpapabilities  with  reference  to  the 
step  (see  example  on  following  page). 

Discussion  pages  which  provide' relevant 
content  and  outline  actions  to  take  in 
applying  the  step. 

At  the  conclusion  of  each  stefe,  refer- 
ences,   introduced* by  the  following  symbol, 


which  may  facilitate  your  implementa- 
tion of  the  step. 
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V 


Sttt 


Explanation  of 
its  purpose." 


ep  and  statement  of 


Definition  of  cohcepts,  knowledge  of 
which  is  central  to  implementing  the 
step  successfully •         /  \ 


jM{  U  .toowledee/C— ablHtte*  Mttn?  C^^~ 

;Revi*w  t&e  information  provided  above.  Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  this  step  on  the  scales  below. 


•  Knowledge 

*  i 

i 

1 

* 

I 

-  Excel- 
lent 

Above 
Avg. 

Aver- 
age 

Below 
Avg. 

Poor 

•Capabilities  - 

1 

• 

» 

Excel- 
lent 

Above 

Avg. 

Below 

Avg. 

Poor 

Based  tfpon  your  ratings,  decide  whether  you  need  to  read  the  content, 
information  on  this  4tep  which  follows  immediately  or  proceed  tct  the 
next  step. 


The  introductory  page — specifically  the/  knowledge  and 
^capabilities  ratjlng  section — has  been  dWeloped  to 
enable  you  to  expedite  your  reading  of  ttte  material 
in  this  section  of  the  Guide.    By  assessing  your  need 
.J  to  know  and  to  improve  on  each  step,  you  can  determine 
*  whether,  or  how  much  attention  and  time  you  have  to 
devoid  to  reviewing  the  information  pertaining  to  that 


step. 


reviewing 

i 


In  addition,  the  back  of  each  introductory  page  has- 
bepn  designed  so  that  you  can  take  notes  and  plan 
action*  that  your  PIC  might  take  to  implement -the 
step  (/see  layout  below)  • 


PME  PROCESS  NOTES/ACTION  PLANNING  PAGE 


MOTE^PoTnTrthatare  liiporf  nt  whicn  you  want  to  remember? 


f^gg^ItS^n^Sucn^o^re  unclear  or  want  to  get  additional 
information)  r 


"ACTIONS  (Specific  a  tape  tnart  you  intend  to  take  to 
T    PIC1 a  PME  proceee  in  fchie  area) 
Stepe  .  Re aoor caa/Par aona 


Implement  your 
Time  Frame 
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A./   ASSESS  LOCAL  NEEDS 


Purpose:       to  determine  the  current  employment  and 
•   training  needs  within  your  community. 


t 


1.  Analyze  Local  Labor  Market  Conditions 
and  Business  Needs  v 

2.  Analyze  Local  Population  Groups  and 
Service  Needs 

3.  Review  Programs  and  Plans  of  Other 
Agencies/Service  Deliverers/Businesses 


ERIC 


III-5 


128 


sup  X 


Analyze  Local  Labor  Market'  Coriaitiqns 


*      1  ** 
The  purpose  of  this  step  is  to  use  labor  marketjtnformation 
to  (1)  identify  potential  job  opportunities  witHin;the  local 
community  nhich  are  appropriate  for,  CETA  participants;  and 
(2)  pinpoint  the'  specific  needs  of  local  businesses  and  indus- 
tries so  that  your  PIC  can  dev&lop  a  plan  based  Upon  these 
factors.     This  step  consists  of  analyzing  the  current  and 
projected  Mature  of  the  l<5bal  labor  market.     Data  to  be  Re- 
viewed include  employment  trends  in  terms  of  labor  force  par- 
"  ticipation;  growth  .poteatial  by  industry  and  occupation;  and 
the  skill  requirements  necessary  for  job  success  in  various 
occupatiqns. 


•  Existing  Labor  Market  Information:     Data  produced  primarily 
by  the  LMI  jflfcvisiOn  of  your  State  Employment  Security  Agency 
(SESA)  whidh  can  be  used  to  analyze-  lo*cal  labor  or  market 
conditions. 

•  Labor  Market  Analysis:  Standard  procedures  to  be  used  to 
analyze  local  labor  market  in  order  to  draw  valid  conclu- 
sions regarding  local  needs.  •  •  , 

•  Employer  Surveys:     Local  surveys  'yhicjh  a  PIC  can  initiate 
in  orderxto  obtain  additional  information  regarding  the 
characteristics  of  the  labor  market  and  the  needs  of 
enfployers. 


Review  the  information  provided  above.  Rate  your  PIC's  knowled^  and 
capabilities  with  reference  to  this  step  on  the  scales  below. 

•  Knowledge 


•  Capabilities 


1  1  . 

I  r 

i 

 1  , 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age  * 

•  Avg. 

>  * 

t  ) 

'I 

 i  

i 

 1 

-Excel- 

' Above 

Aver- 

Below 

Poor 

lent 

AvA. 

age  r 

Avg. 

Based  upon  your  ratings,  decide  whither  you  'need  to  read  the  content 
information  on  this  itep  which  follows  immediately, or  proceed  to  the 
next  step.(  ^  -  \  • 
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PROCESS  NOTES/ ACTION  PLANNING  PAGE 


NOTES  (Points  that  are  important  which  you  want-  to  remember) 


QUESTIONS  (Items  on  which  you  are  unclear  or  want  to  get  additional 
•  information) 


ACTIONS  (Specific  steps  that  you  intend  to  take  to  implement  your 
"*      7         PIC's  PME  process  in  this  .area)  ' 

Time  Frame 


Steps 


Re  sources/Persons 


/ 


i 
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INTRODUCTION 

Labor  market  information  (LMI)  is  data  on  the  labQr  fo?ce> 
occupations,  and  job  openings  and  hiring  procedures  within 
your  local  community.    You  should  use  this  inforjnation  - 
especially  ihat  on  job  opportunities  -  to  help  determine  thS' 
overall  structure  and  content  of  your  PIQj.s  PSIP  and  to  help 
meet  the  business  needs  in  your  community.'    In  this  regard*  you 
can  use  LMI  for  a  .variety  of  purposes,  including: 

•  selecting  target  jobs  for  training  pro- 
grams 4 

•  '  designing  customized  training  programs 
to  satisfy ^locally  defined  criteria  j 

r 

•  .identifying  obstacles  and  barriers  which  ' 

inhibit  the  effectiveness  of  employment*  '  's 

and  training  programs  so  that  these  . 
impediments  can  be  eliminated 

•  determining  services  (e.g.,  employment 
development,  woifld  of  work  orientation,  3 
recruitment)  which  can  be  provided 

through  the  PIC  t6  meet  local  employer  * 
needs  *  ^  t 

•  instituting  other  'special  programs  to 
improve  the  relationship  between  pr^vatfe 
Sector  employers  and  your  PIC's  progrartL  ' 

O       \  : 

You  can  take  one  of  two  approaches  to  labor  market  analysis, 
You  can  rely  solely  on  existing  data  sources  (i.e.,  that^fur- 
nished  by  the  LabtfE^-MarkeJt  Information  Division  of  your  State  - 
Employment,  Security  Agency  (SESA) ,  the  Prime  Sponsor,  and  other 
governmental  agencies) •    Or,  you  can  complement  «the  available 
data* with  a  local  employer  survey  of  your  own.     Regardless  of  . 
the  approach\our  PIC  chooses,  by  bringing  the  private  sectpc  / 
perspective  to  bear  in  the  analysis,  you  can  enhance  the  utility 
of  labor  market  information  both  in  your  PIC's  program  and' in 
the  CETA  program  in  general* 
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Existing  data  sources  will  probably  be  sufficient  to  help 
you  determine  general  trends  and  answer  many  basic  questions 
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regarding  local 

# 

.  •  labor  force  participation  - 
*  #  growth,  potential  fcy  industry  and  occupation 

•  salary  range  and  quality  of  occupations 

•  skill  requirements  necessary  for  job  suc- 

•  cess  in  various  occupations* 

However,'  existing  data  may  be  inadequate  to  answer  the  questions 
you  feel  are  moyt  important  'regarding  the  local  labor  market. 


Local  Employer  Surveys  - 

"  9 


Local  employer  surveys  enable  you  to  determine  a  number  of 
additional  things  regarding "the  local  labor  market,  including: 

•  nature  and  structure  of  the 'internal  labor  A 
market' (job -composition  and  entry  level 
occupations)  of  local  firms 

•  dumber  and  occupational  characteristics 
of  existing  job  vacancies  in  local  labor 

'   *  markets  ,  / . 

•  hiring  practices  and  policies  of  local 
firms        *  1 

^         •  . 

•  hourly'  and/or  weekly  wages'  and  annual 

-   earnings  6f  workers  in  specific  occupa- 
tions  v  * 

•  training  and  promotional  policies  of 
local  firms 

•j^hployer  attitude^  toward  existing  employ- 
ment and  trsining^programs  funfied  by  the 
*  local  CETA  prime  sponsor  and  other  local 

employment  and  training  agencies.  4  -» 

V)  ■ 
Your* choice  of  how  to  do'labor  market  analysis  will  be 

determined  by;  **  * 

•  the  adequacy  and  accuracy  of  the  data 
available  to  you 

•  the  purpose  you  want  to  achieve  with 

your  analysis        ~      *  * 

•  the  decisions  you,: want  to  be  able  to  make. 

§ince  local  employer  surveys  are  time-co^sumipg  and  require 
considerable  knowledge  and  Expertise  for  their-proper  implementa- 
tion'/ you  should  carefully  review  your  data  before  ini^iating'^ne 
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The  remainder  of  this  section  provides  information  on:' 

*  « 

•  .how  you  can  secure  and  use  existing  LMIj 
and  * 

#  the  considerations  to  be  taken  into  account 
in  deciding  fcpon  and  implementing  your  own 

- local  employer  survey .  . 


A.     EMPLOYING  EXISTING  LMI 


i  » 

Your  first  step  toward  doing  a  Labor  Market  analysis 
should  be  to  obtain  all  existing  data  and  information. 

Where  to  Get  LMI  .  '  * 

1  Within  the  U.S.  Department  of  Labor,  the  LMI  network  con- 
sists primarily  of  an  interagency  working  arrangement 'between* 
the  Employment  and  Training  Administration'  (ETA) ,  the  Bureau 
of  Labor  Statistics  (BLS) ,  and  each  State  Employment  Security 

t 

Agency  (SESA) .  •  ■ 

ETA  plays  a  major  role  in  planning  and  funding  LMI  activi- 
ties.   The.  SE^A  LMI  division  has  the  responsibility  for  de- 
velopment and  dissemination  of  LMI.  -  The  BLS  provides  tHfe 
technical  support  and  guidance  to  carry  out  cooperative  statis- 
tical programs  (employment,  unemployment,  occupational  projec- 
tions,, labor  turnover  statistics,  and  employment  and  wages  in 
firms  covered  by  the .unemployment  insurance  program). 

Your  PIC  should  begin  its  search  for  LMJ  by .contacting 
your  Prime  Sponsor  to  get  all  available  data  and  then  con- 
tacting the  LMI  Division  of  your  SESA.    Most  SESAs  have  LMI  _ 
analysts  in  the  state  office  and  in  each  district  o'ffice.  Both 
can  provide  LMI  as  well  as  other  assistance.  N 

As  your  first  order  of  business,  you  should  request  an 
LMI  Directory  or  Bibl iography  of  Sources  in  order  to  become 
\    familiar  with  the  .vajrfcety  of  available  documents.    The  SESA 
-  analysts  can  "alstTguide^you  to  specialized  data  or  information 
,  available  from  other  sources."    Since  data  are  available  from 
many  sources,  the  SESA  analysts,  as  both  experienced  data 
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producers  and  users,  and  because  of  their  proximity  to  your  Pic, 
can  often  provide  n6t  only  the  data  but  also,  valuable  suggestions 
on  what  data,  to  get  and  how  to  analyze  and  interpret  the  LMI. 
For  tfhese  reasons,  if  your  PIC's  data  needs  go  beyond  the  scope 
and  resources  of  your  SESA' s- existing  LMI  program,  you  might 
.considW^o^iding  additional  funding  to  your  SESA's  LMI  division 
for  specialized  research  and  data  development.  . 

Yo\i  should  not,  however,  restrict  your  search  for  existing 
data  to  £he  SESA.    You  should  also  contact  the  LMI 
specialists  in  ETA's  Regional  Office  and  the  staff  in  the  BLS 
•Regional  Office,    within  the  DOL  network,  there  may  be  specialized 
data  or  new  programs  of  which  the  SESA  analysts  are  unaware.  The 
ETA  and  BLS  regional  offices  have  access  to  these  recent  develop- 
ments and  can  provide  data  to  complement  the  existing  SESA 
information.      finally,  you  should  also  contact  your  State/*. 
Occupational  Information  Coordinating  Committee.     The  SOICC 
is  responsible  for  coordinating  the  development  of  occupational 
information  and^can  guide  you  to  other  potential  sources  of 
relevant  data.  4 

Hdw  to  Use  Existing  Data 

How  should  your  Pic  use  existing  LMI* data  once  you  secure 
it?    Existing  data  is  probably  best* suited' for  identifying  poten- 
tial job  o^anings' within  the  local  community.    To  identify  more 
openings  in  a  sound  man<ner,,$bu  should  know  the  principles  under- 
lying the  use  of  and  the  general  process  for  using  LMI  data. 
Then,  you  should  be  able  to  apply  these  principles' and  the  \ 

process  in'  a  ^tep-by-step  fashion  to  your  local  situation. 
r  m 

4 

!•    Principles  and  Process  of  LMI 

The  basic  principle  underlying  the'mse  of  LMI  is  that-  job 
openings  can  bef  expected  to  be  greatest  among  those  ke£  indus- 
tries and  occupations  within  the  local  community  which  are 
experiencing  large  growth  and  replacement  needs.  Concepts 
which  ypu ^hould  understand  that  are  central  here  include: 

industries/businesses  and  emplo/ers 
within  the  local  community 

'III-12 


134 


r 


A 


•  occupations/ various  job  categories 

within  those  industries 

►  -  ' 

•  job  openings/ vacancies  within  specific 
occupations  *  % 

%  •  employment  growth/net  additions  t6 

the  number  of  workers  needed 
»  * 

•  replacement  needs/openings  caused  by 
workers  leaving  jobs. 

The  general  process  for  using7 fflese  concepts  to  identify 
potential  job  openings  within  your  local  community  is  as 
follows: 


Identify  specific  industries  within 
your  community  which  you'  think  might 
have  a*  potential  for  job  openings. 

Identify  specific  occupations  asso- 
ciated with  those  industries. 

Use  growth  and  replacement  -need  data 
tq  pinpoint  specific  industries  and 
occupations  in  which  job  openings 
seem  most  likely.  ^ 

For  each  potential  occupation,  measure 
job  quality  and  placement  potential 
by  using  available  LMI  on  training  and 
education  requirements/  expected  wage 
rates,  and  supply  of  workers  which 
might  pOssibly  compete  for  job  open- 
ings, v  s 

After  you  have  identified  specific 
suitabler  occupations  with  potential 
openings,  you  contact  local  employers 
in  order  to  verify  that  opportunities  # 
do  exist. 


2.     Steps  in  Labor  Market  Analysis 

♦ 

A  more  detailed  explanation  of  the  steps  your  PIC  should 
±ake  to  use  local  data  in  identifying  job  openings  follows. 


a.     Identify  tHe  geographic  lab6r  market 

The  initial  ?tep  in  the-^ob  identif icati'ori~li*a$ess  involves 
identifying  the  geographic  jurisdiction  of  your  *>IC's  local 
labor  market.    The  area  of  immediate  interest  may  be  your  Prime 
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Sponsor  jurisdiction.  ~Howe\ter,  you  should  not  make  your  focus 
€00  narrow.    You  should  attempt  to  look  at  job  opportunities  > 
fully  within  the  context  of  the  geography  for  economic  activity 
that  generates  employment  -  that -context  may  be 'a  geographic 
area  somewhat  different  from. your  Prime  Sponsor's  jurisdiction. 

9  In  identifying  your  PlC's  geographic  labor  market,  you- 
should  consider  the  Labor  Market  Area  (LMA)  and  the  Standard 
Metropolitan  Statistical,  Area  {SMSA)  .    These  two  arSas  are 
based  upon  concepts  of  economic  activity  rather  th%i  political 
boundaries.    Thus,  there  are  logical  geographic  building  blocks 
for  labor  market  analysis.     Both  the  LMA  and  SMSA  are  defined 
by  the  relationship  between  place  of  residence  and  place  of  work 
of  the  labor  force.     The  primary  consideration  in  their • defini- 
tion is  the  ability  of  labor  to 'accept  new  .  jobs  without  the 
necessity  of  changing  residences  or  incurring  unreasonable 
commuting  distances.  •  \, 

To  $void  fragmented; 'incomplete'  interpretations  of  job 
activity,  as  a  rule, your  analysis  should  first  be  conducted  at 
the  LMA  or  SMSA  geographic  level.     Then  you  shbuld  reduce  yoltr 
focus  to  your  local  arerf.    -If  you  are  uncertain  abcjut  what  your 
PIC's  geographic  labor  market  should  be,  then  contact  the 
local  SESA  analyst  or  LMI  Specialist  who  can  provide  assistance 
in-determining  an  adequate  geographic  jurisdiction  for 
analysis.  .  *  • 

Identify  the  industry  structure  of  employment 
Having  identified  the  geog^afShic  area  for  analysis,  "your 
next  step  is  to  become  familiar  with  the  industrial  composition 
of  the  area',  the  employment  levels  among  industries,  and  the 
employers  among  industries.     The'size  of  an  industry  and  the 
number  of  employers  within  an  .industry  can  serve  as  an  initial- 
guide  to  key  industries.  0 
For  exatople,  because  of  lattor  turnover  and  worker  separations, 
openings  will  occur  among  several  industries.     If  separation  * 
rates  are  uniformly  low  or  assumed  to  be  at  a  minimal  level, 
the  largest  number  of  openings  should  occur  within  the  largest 


111-14 


el  36 


,,indu3£rial  categories.     Stated  simply,  most*  of  the  job  \ 
openings*  will  oe^yr  wherfe.most  of  the  jobs  aye  . located.  Be- 
cause  of  the  voiume^of  employment. in  large  industries,  separa- 
tions, wi'lL* creafe^T^^rge  number  of"  openings.     From  that  view, 
large  industries  or  .industries  with  large  establishments  might 
bte  considered  key  leading  iudustries. 

<*    From  a  second  perspective,  however,  your  PIC  should 
understand  that,  in  certain J^iflPfeances,  smaller  firms  generate' 
mpre  new  jobs.  ^T^is 'is  highly  "dependent  upon  the  structure  and 
nature  of  the  taocal  acpnomy,.    Thetefqre,  your'  PIC  should  also  , 
closely  .examine  the  local'  industrial  structurerto  identify 
.industries  concentrated  with  several  smaj.1  firms  to  identify 
potential  leading  industries.    This  is  especially . impottarfll    *  . 
'given  your  PiC's  mandate  to  secure  greater  involvement  of  ^ 
.  small  businesses  in  solving'  employment"  an&<^dltiing  problems. 
/  A  varj^ty  of  available;  LMl\:an  ;H^>used  ^^a'naLyze  indus- 
trial ^ircucture.    These  sources  include  the  data^trom.  Employ- 
ment;Wages  and  Contributions  of  Employers  Cov^fed  by  Unem-  - 
ployraent  Insurance  (ES-202)  ,  790  or  -Current  Employment  Statis- 
Lcs  data,  or  infoipnatipn  from  County  Business  Patterns. 


c.    Analyze  historical  employment  TrenqLg^ 
*      After  you:  become  fami^i^r  with  the  strufctiSre  of  lQcal 
^industrial  employment  within  your  PIC's  area,  the  next  step, is 
to  study  employment  trends^.    Specifically,  it^i*-4jnportant  to 
J^now  where  growth"  has  ofccurred  in  the*  past  for  purposed  of 
projecting  continued  growth  trends.    To  determine  this,  you 
select  an*  analysis-,- base  and  target  years  that  seem  to  best 
fit  the  ttend.of  total  industrial  employment  over  the*  past 
f  ive' years..  <  (Then,  you  do  simple  calculations  to  identify  the 
absolute  and  percentage  change  in  employment  by  industry. 

By  exsfmihlnCi  the  historical  trends  of  each  industry  !s 
employment,  you  6an  idtentify  which  industries  have  experienced 
tjie  greatest  absolute  growth  orj^the  fastest  rise#(percentagd 
irifcrease)  in  employment.     Conversely/  you  can  pinpoint  indu?-' 
'  tries  which  have  large  number*  of  employees,  yet  are' actually 
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declining.    .{-These  determinations  can  be  made  usin^a 'table 
similar  to  the  one  in  fixhibi-t  1  on  the  following  page  J 

Interpret  current  economic  conditions  of  area  industries 
Although  it  is  necesgary  to  see  where  growth  has  occurred 
^3^e  past,  it  Is  equally  impottant  to  determine  how  those  indgs 
tries  have  been  functioning  in  the  more  retfent  time  Reriod. 
good,  plaice  to  begin  is  with  the  local  labor  market  information' 
newsletter  published' by  the  local  SESA.    The  newsletter  pror 
vides  narrative  .analysis  of  Current  Employment  Statistics.  (CES) 
on  employment  and  the  number  of  r^nagricultural  wage  and 
salary  jobs  by  major  .two-digit  (SIC)  industry.     By  comparing • 
the  figures  for  the  recent  month  and  for  the  same  month  in  the 
previous  year, "you-can  obtain  an  insight  into  how  industry 
employment  has^  expanded  or  contracted  over  the  past  year.* 

•  * 

e.     Inspect'  other  current  dkta      "  . 

There  are  other  valuable  data*  items  which' you 'can  consult 
to  exaunine  recent  "industrial,  trends,  such  as  labod  turnover  ■ 
rates  and  Employment  Service  job  openings^  Gomr^es  include: 

•  Data  from  the  Labor  Turnover  Statistics  (LTS) 
program.  ><Clthough  restricted  to  manufac-  ~" 
turing-^ninirtg,  and  communications,  the  LTS 
-projfitfes  information  on  labor  turnover  rates 
.utrflroyigh  a  monthly  employer  survey  normally 
incorporated  in£o  a  monthly  newsletter.* 


•  fes  Job  Bank  and  ESARS  (Employment  Security 
Automated  Reporting  System)  tables  to  iden-  , 


ti^v.current  openings  registered  with  the 
job  service  by  industry  to  determine  if"  . 
there'  is  a-demand  for  workers  in  those  \ 
industries.    The  Job  Bank  and  ESARS  indi- 
cate how  many  openings  are  listed  with  the 
ES,  in  which  two-digit  industries  these 
openings  occur,  and  how  many  remained  open 
for  30  days  ot  more.    Thfey  *1jm>  provide 
some  wage  and  hiring  ^fprmrffT^n 


/    t  f.     Incorporate  employment  projections  in  the  analysis 
N     After  you  have  examined  the  current  industrial  structure, 
and  recent  and  historical  trends  by  industry,  you  should  look^ 
at  industry  employment  projections  to  obtain  a  complementary 
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EXHIBIT?  I 


CHANGE  IN  INOjUStRIAL  STRUfcTURE  OF  EMPLOYMENT 
DALLAS  COUNTY,  TEXAS 
1972-1977 


SIC 


Industry 


Total  Covered 
Employment 
1972  1977 


Change  In 
Employment 
1972  1977 
IMumb»f  percent 


(A) 


• 

•  •• 
• 


20 
22 
23 
24 
25 
26 
27 
28 
29 
30 
31 
32 
33 
34 
35 
36 
37 
38 
39 

•  •• 

•  •• 
• 

•** 

•  ** 

•  *! 


(8)  W 

Total 

Agricultural  Services 

Mining  „ 

Contract  Construction 

Manufacturing 

Food  and  Kindred  Products 
Textile  Mill  Products" 
Apparel  and  Other  Textile  Goods 
Lumber  and  Wood  Products 
Furniture  and  Fixtures 
Paper  and  Allied  Products 
Printing  and  Publishing 
Chemicals  and  Allied  Products 
Petroleum  and  Coal  Products 
Rubber  and  Plastic  Products 
Leather  ami  Leather  Products* 
Stone.  Clay  and  Glass  Products 
Pnmary  Metal  Industry' 
Fabricated  Metal  Products 
Machinery.  Except  Electrical 
fleet nc  and. Electronic  Equipment 
Transportation  Equipment 
*  Instruments  and  Related  Products 
Miscellaneous  Mfg  Industries 

Transportation  and  Other  P  U 
Wholesale  Trade 

Retail  Trade  a  j 
Finance.  Insurance  and  Real  Estate  (-  / 
Services 

Nonclassifiabie  Establishments  *     j  % 


(C) 

593.964 

1.248 
7.748 
45.8g2 


48.024 
64.638 

1 12.686 
59.799 

103.059 
3.461 


55.911 
70.056 

135.993, 
72.915 

137.180 
1.461 


1642 
8  38 

20  68 

21  93 
33  11 

-57.79 


Source:  U.S.  Department  of  Commerce.  Bureau 
Payroll  and  Establishments,  by  Industry 
and  1977). 


of  the  Census.  "Table  2.  Counties — Employees. 
County' Business  Patterns:  Texas.  (1 972 
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perspective  of  'the  industrial  structure    Employment  projections 
provide  employment  data  by  industry  for  a  base  year,  a  current 
estimated  year,*and  a  target  year  so  that  it  is ^possible  to 
calculate  absolute  and  percentage  change,  thereby  identifying 
where  the"  greatest  amount  of  industrial  growth  is  expected  to 
Occur.     Another  valuable  source  to  consider  is  the  industrial 
projections  i,n"the  Industrial  Outlook  Handbook.     This  Handbook 
provides  a  ^cQm^ct  survey  of  U.S.  business  and  an  overview  of 
the  development  which  will  influence' their  growth  potential. 
Other  potential  spurces  of  data  include  employment  reports  by 
local. chambers^ of  commerce,  .economic  development;  organizations, 
and  othef  |lQcal  private  sector  or  governmental  agencies. 


g. 


Rank  rbccil  industries  according  to  their  employ 
men.t  potential  * 

*    ^  After  yop'lre  jfcmpleted  your  general  analysis,  you  should 
identify  industries  which  h^ve  the  greatest  potential  for 
employment  groWth/    This  will  allow  you  to  narrow  the  exten- 
sive local  -  ihdjpsferial  structure  down- to  a  few  key  leading 
i^^t^i^^hich  ;c$n  be  more  easily 'analyzed.    As  part  of  this 
J^c|^>  youK should  narrow  the  industries  $ven  further  by 
id^^ifyirCg  the  .specif ic  thre£  or  four-digit  (SIC  code)  ihdus- 
tr^e^^fcidch  experienced  growth  and  are  projected  to  continue 
thelil  gr^wijh.     An.  important  point  "  to  remember  here  is  thatt 
in  an)aitalysis  qf  available  LMI,  there  normally  will  bl\a 
limited  nurafcrt^bf  induq^ies  that  consistently  stand  out  as 
being;  fheal,thy  wittw^ood  growtji  potential  and  a  high  demand  for 
wdr-kers  due  £0  growth.  * 


9 

*  h 


tyl     identify  local  employers 
*    Next,  you  should  refer  to  a  local  Directory  of  Manufac-  / 
turers ,  a  Dun  and  Bradstreet  listing. of  million  dollar  or  -  m 
mi<%ile  level 'firms, „ or  a  Chamber  of  Commerce  listing  of  local 
businesses  to  get  names,  addresses  and  marketing  scop^Nof 
individual  firms  within  those  industries.  *  By  integrating  ttji^' 
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statistical  overview  of  the  sources  of  industrial  employment 
with  'specif ic  information  on  the  local  employment  structure, 
you  can  identify  individuals  within  high  priority  potential 
industries  whom  you  may  wpnt  to  contact  >as  part  of  y^ur  local 
labor  market  analysis.  - 


#  » 


i.    Analyze  the  occupational  composition  of  selected 
industries  .  „  : 

After  you've  picked  industries,  you  need  to  .identify  the 

occupations  (in  relative  terms)  that  comprise  the  work  force 
♦ 

for  each  selected  industry.'  The  questions  -that  need  to  be 
answered  concerning  occupations  are:  — 

S  .         •  How  many  jobs  are  there  in  each  occu- 
pation within  the  key  industries? 

•  What  occupations  have  the  largest  number 
of  jobs  in  those  industries? 

•  In  which  occupations  are  future  job 
opportimities  most  likely  to  ot:cur? 

The  occupational  composition  of  employment  within  iftdus- 
tries,"  taken  from  the  Occupation  Employment  Statistics  (OES) 
susvey  program  done  by  SESA,will  help* determine  which  occupa-  • 
tions  make  up  employment  within  a  particular  industry  and  in 
what  percentages  they- are  distributed.    Although  industry 
staffing  patterns  may  be  available  at  the  s.tate  level  only, 
it  may  be  feasible  to  apply  those  same  staffing  patterns  to 
the  local  area.    Staffing  patterns-  will  provide  percentage 
distributions  of  major  occupational  groups.    By  tabulating 
those  occupations  by  firms  witfein  an  industry,  you  can  iden- 
tify the  roost  commonly  reported  occupations  within  the  industry. 

j.    Analyze-^occ^pational  projections 

1  While  it  Should  W  recognized  tha*  occupational  projec-  , 

tions  have  limitations^  they^/e  invaluablV^iti  identifying 

,growth  occupations  by^irt£ustry.    Although  some  states  are  still 
using  census-based  projections,  many  states  now  have  projections 
Available  from  the  OES,  st*vey  program;    The  OES  projections  are 
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-derived  by  multiplying  total  employment  projections  for' a  target 
'year  by' industrial  staffing  patterns.    To.  ascertain  local 
needs  you  shouid  employ  that  daj:a  which  is  available  to  you. 

k.     Consider  job  quality 

However,  you  should  not  restrict  your  analysis  to  only 
the  number  of  jobs  potentially  available  within  an  occupation. 
To  facilitate  the  best  match  between  workers  and  jobs,  you 
should  assess  the  quality  of  available  jobs. 

To  accomplish  this,  you  should  analyze  the  nature  of  the 
job  according  to: 

•  transferability  of  skills  within  that 
occupation 

•  job  duties  and  functions 

*  •  basic^hiring  requirements  (including 
ports  of  entry)  for  placement  into 
that  occupation  * 

r 

•  pay  range  or  average  wage  rate 

•  firm-specific  training  requirements 
of  potential  employes. 

Considering  job  quality  adequately  consists  of  two  parts: 
reviewing  and  analyzing  available  LMI,  and  gathering  additional 
Nqualitative  information  through  coi^acts  with  local  employers 
to  supplement  available  dfta. 

.    To  assess  30b  quality,- generally  you  can  consult  the  local 
area  nonseasonal,  occupational  shortage  (hard-to-fill)  listing 
6r  JOB-FLO  data  provided  by  the  SESX.     You  can  isolate  occupa- 
tions which  have  transferable  skills  by  referring  to  the  basic 
duties  and  functions  associated  with  occupational  categories 
in  occupational  classification  manuals,  such  as: 

•  Dictionary  of  Occupational  Titles  (DOT). 
The  DOT  includes  general  tasks  involved 

in  the  occupation,  fields,  of  specializa-  , 
tion  within  an  occupation,  and  statements 
**  which  descril^e  duties  required  of  workers 
in  this  occupation 

*» 

•  Standard  Occupation  Classification  Manual  % 

(sop .  :  ~  .  • 
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Finally,  you  can  employ  the  JOB-FLO  to  acquire  information 
on  unfilled  openings,  pay  range,  ye^rs  of  education  required, 
and  months  of  experience  required  for  employment.    (These  data 
are  collected  from  job  listings  by  employers  with  the. Employ--  - 
ment  Service.     Unfortunately,'  JOB-FLO  publication  is  restricted 
solely  to  those  occupations  in  the  local  area  which  are  listed 
with  the  Employment  Service.  1 

To  get  specific  information  on  job  quality  and  training 
requirements,  there  is  no*  substitute  for*  personal  contact 
with  local  employers.    Since  the  majority  of  actual  training 
takes "place  on  the  job,  by  contacting  the  individual  employer, 
you  can  learn  what  types  >/Of  ^training  are  required  to  place 
^workers  in  the  .firm's  entry-level  position.    This  information 
can  be  used  to  structure  PIC  programs  to  satisfy  hiring  pre- 
requisites.   This  greatly  increases  the  possibility  of  place- 
ment after  training.  \ 

)    In  conclusion,  existing  data  and  sources  can  be  extremely 
helpful  in  deciding  what  programs  and  services  the  PIC  should 
provide/    However,  personal  contact  "always  enhances  the 
specificity  of  and  potential  Receptivity  for  your  programs. 


iB.  .  DOING  LOCAL  EMPLOYER  SURVEYS 

Your  existing  LMI  dd^a  may  be  sufficient  to  help  you 
target  your  PIC's  program.^  However,  in  those  cases  when  you 
,feel  that  there  are  gaps  in  existing  data  or  you  need  to  know 
local  needs  more  precisely,  the  best  thing  to  do  is  to  contact 
local  employers  personally  for  job-specific  information. 
Through  a  carefully  constructed  and  conducted  survey  of  local 
employers,  you  can  secure  invaluable  data  which  your  Pip  can 
use  in  conjunction  with  existing  LMI  to  improve  the  planning, 
design,  and  implementation  of  its  total  program.    The  discussion 
which  follows  highlights  information  And^  provides  guidance 
related  to:    uses  of  local  employer  surveys;  making  the  employer 
survey  decision;  and  implementing  the  employer  survey*. 
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J-The  nature  o^LMI  data  available  varies  considerably  from  state,  to  state.  For 

;  in  the 
Turnpve 

143 


exasple'*  several  states  do  not  participate  in  the  Occupation  Employment  Statis- 
tics (0ES)  survey  program  or  in  the  Labor  Turnover  Survey  (LTS)  . 
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Uses  of  Employer  Surveys 

/   You  can  use  employer  surveys  to. determine  a  number  of 

things,  including  the:  ? 

,*  » 

•  nature  and  structure  of  the  internal  labor 
markets  of  local  fifms,  including  those 
not  Tinted  with  the  ES 

•  number  and.  occupational  characteristics  of 
existing  job  vacancies  in  local  labor .markets 

•  hiring  practice*  and  policies  of  local  firms  ' 

•  hourly  and/or  weekly  wfcges  an<J  annual  earnings 

of  workers  in  specific  occupations  f  % 

•  training  and  promotional. policies  of 
local  firms  . 

•  employer  attitudes  toward  existing  employ- 
ment and  training  programs  funded  by  the 
local  CBTA  Prime  Sponsor  and  other  local 

-   employment  and  training  agencies. 

✓ 

* 

1.     Internal  Labor  Markets  and"  Entry-Level  Occupations 

Knowledge  of  the  nature  And  structure  of  the  internal  labor 
markets  of  local  firms  in  potential  high  oppdrturilty  job  opening 
areas  can  enafcle  you  to  plan  highly-focused  occupation^lly- 
oriented  training  programs  to  be  run  by  your  PIC. 

The  jobs  within  a  firm's  internal  labor  market  consist  of 
entry-level  jobs  which  are  open  to  persons  outside  the  business, 
'and  thos6  jobs  tied  to  internal  career  ladders  that  are  filled 
by  the  movement  (promotion  or  transfer)  of  incumbent  employees* 
Knowledge  of  the  specific  occupations  that  comprise  the  ports 
of  entry  within  local  business  establishments  and  their  linkages 
with  other  jobs  in  the  establishment  can  enable  you  to  target 
your  PlCrs  occupational  classroom  draining  programs  toward  those 
"entry-level"  jobs  which  will  be  accfessible  to  successful  program 
graduates.     In  addition,  ^ou*  "ca^  use  the  "knowledge  that  you  gain 
regarding  entry-level  jobs, that  pay  relatively  low  wages  and 
provide  few  promotional  opportunities, to  identify  areas  in  which 
your  PIC  might  implement  upgrading  programs.     Or,  you  can^use 
information  on  contracting  and  expanding  occupations  to  identify 
possible  targets  for  retraining  efforts. 
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2.    Job  Openings  Data 

You  can  use  data  on  the  cjuantity  and  the  industrial  and 
occupational  characteristics  of  current  job  Openings  in  local' 
markets  tor  a  number  of  diffefent  program  planning  purposes- 
Data  on  the  number  of  current  job  openings  by  occupation  can 
be  utilized,  together  with  data  on  *he  numbers  of  unemployed  * 
workers  by  occupation,  to  identify  occupational  areas  charac- 
terized by  current  shortages  in  the  local  labor  market.  Some 
of  these  occupations  could  be  desirable  candidates  for  the 
provision  of  classroom  training  under  Title  VII. 

^lowever,  not  all  such  shortage  occupations  would  be  suitable 
candidates  for  training.    Relatively  low  wagea^nd  adverse 
working  conditions  may  be  responsible  for  the  existence  of  *some> 
occupational  shortages.'    Such  occupations'  are  frequently  char- 
acterized^ by  high  rates  of  voluntary  turnover  and  may  not  pro- 
vide the  employment  stability  and  earnings  necessary  to  achieve  ^ 
longer-term  employment  goals.  .  % 

By  distinguishing"""" less  desirable"  occupational  shortages  . 
areas,  you  can  then  design  your  PIC' s .program  to* concentrate 
on  preparing  unemployed  and  economically  disadvantaged  persons 
for  employment  in  "desirable"  occupations.    This  will  enable 
you  to  meet  simultaneously  the  needs^of  private  sector  em-  s 
ployers  and  the  unemployed  and  have  a  favorable  net  impact 
upon  reducing  the  level  of  unemployment  within  your  community. 

If  your  PIC  chooses  to  do  a  joF  opening  type  survey,  it 
would  be/Best  to  focus  upon  subsets  of  local  industries  char- 
acterized by  recent  abdjve  average .  rates  of  growth  in  employ- 
ment and  by  staffing  patterns'  containing  occupations  that  have  • 
substantive  skill  ^requirements  for  which  training  can  be  pro-  * 
vided  to  participants  in  a  reasonable  period  of  time  (omentum  of  6-8 
months) .     Getting  this  type  of  data  will  enable  you  to  reach 
training  decisions  based  not  only  upon  the  availability  .of 
current  job  openings  in  an  occupation ,  but  also  upon  knowledge  e 
•of  the  ability  of 'training  programs  to  prepare  CETA-eligible 
individuals  successf ully ^fqr  such  job^  with  the  likelihood  of 
the  jobs  being  open  to  graduates  upon^heix  completion  of  the  £ 
program.  /^^^t       —  ^ 
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3.  Hiring  Policies  and  Practices 

You  can  also  design  your  local  employer  surveys  collect 
a  wide  variety  of  information  on  the  hiring  requirements  for 
particular  occupations  and  the  sources  of  labor  supply  utilized 
by  various- local  %f  inns  in  securing  their  needed  manpower  in 
those  same  occupations.   .Knowledge  of  the  formal  education, 
experience,  and  specific  skill  requirements*  for  particular 
^occupations  as  well  as  personality  traits,  test  scores, '  licen- 
sing,  and  ot^ier  requirements  of  local  employers  secured  through 
such*  surveys  can  then  be*\^fed  to  determine  appropriate  curri- 
culum 'content  for  classroom  training  programs,  and  criteria 
for  the  selection  of  CETA-eligible  individuals  for  partici- 
pation in  these  programs.. 

4 .  Information  on  Hourly  Wages  and 
^     Hours  of  Work  by  Occupation 

One  of  the  'key  characteristics  of  occupations  that  should 
he  taken  into  consideration  by  your  RIC  in  selecting 'occupational 
areas  for  training* are  their  prevailing  hourly  wages  and  weekly 
hours  of  work.     In  the  1978  reauthorization  of  the  CETA  legis- 
lation, Congress  declare^  that  the  employment  and  training 
services  provided  to  participants  under  the  Act  should  ulti- 
mately lead  to  an  increase  in  their  earned  incomes  and  assist 
them  in  becoming  economically  self-sufficient. 

There  are  several  l£cal  data  sources  available  that  provide 
some  information  on  the  hourly  or  weekly  wages  of  workers  in 
particular  occupations,  including  SESA  ESARS,  Job  Bank  data, 
and  the  Bureau  of  Labor4  Statistics  Area  Wage. Srurveys .  The 
available  data  sources  on  wages  of  workers  in  specific  occu- 
pations tend  to  have  a  number  of  limitations^  however,  including, 
their  limited 'occupational  Coverage  and^Jimited  degree  of  repre- 
sentativeness.    Therefore,  if  you  cantiot  obtain  " good n4  hourly 
wages  and  work  hours "information  on  occupational  areas  in  which 
your  PIC  is* considering  funding  programs,  you  might  want  to  de- 
sign your^own  survey  focused  upon  these  areas. 
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A  survey  of  this  type  should  attempt  to  ^pture  informa- 
tion on  the  starting  hourly  and/or  weekly  wages  (no  previous 
experience)  for  workers  in  an  occupation,  the  average  wages 
being  earned  by  workers  in  these  occupations,  and  the  maximum 
wages  that  workers  can  earn  in  this  occupation.     Data  on  the 
weekly  hours  of  work-^n  the  occupation  should  also  be  collected 
during  the  survey \s  wpll  ^s  data  on  the  typical  eraploymerfC' 
stability  of  these  jobs.     The  wagQ,  and  hours  data  for  eacii 
occupatiori  should  be  analyzed,  to  determine  the  variations  in 
wages  paid  to  workers  ampng  local  industries  and  among  firms 
of  different  employment  size  classes  within  a  ggivwi  major  in- 
dustry group. 

*  Your  PIC  can  use  the  knowledge  of  these  local  interindustry 
and  inter firm  wage  and  earnings^  differentials  for  given  occu- 
pations in -guiding  the  development  of  OJT  contracts  for  CETA- 
eligibie  individuals,  and* -the  job  placement  process  for  gradiittes^ 
of  future  classroom  training  programs  in  these  occupational 
areas. 

5.     Training  Policies  and  Promotional  Opportunities 

Another  potential  focus  for  an  employer  survey  is  upon 
training  policies  and  the  internal^pgomotional  opportunities 
.of  local  firms.    Surveys  of  ttfls  type  should  ideally  be  / 
limited  tp  a  relatively  small  subset  (10-20)  of  occupations 
being  considered ♦for  training.    Many  firms  providejpost-employ- 
raent  or  on-the-job  training^(OJT) . 

By  surveying  the  local  firms,  and  occupations  toward  which 
you  intend  to  direct  your  programs,  ypu  can  ascertain  the  em- 
ployer's perceptions  of  the  types  of  skills  that  workers  need 
to  bring  with  thenT  to  the  jofc  to  perform  in  a  satisfactory 
manner  and  information  on  the  types  of  skills  that  they  (em- 
ployers) will  impart  to  the  workers  through  information  and, 
-structured  OJT.    Xou  can  then  design  the- curricula  for  your  , 
classroom  training  programs  to  address  specifically  the  pre- 
employment  needs  4s  expressed  by  potential  employers. 
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In  addition,  you  can  use  this  form  of  survey  to  acquire 
information  on  the  degree -and  nature  of  internal  promotional 
opportunities  available  to  worfters  in  the  occupational  area 
in  which  your  PIC  is  interested.     These  findings  on  promotion 
opportunities  can  then  be  utilized,  togetfce*  with  other  in- 
formation on  hiring  requirements,  training*  policies,  beginning 
wages,  anA  employment  stability  of  entry-levfel  occupations,  in 
determining  those  occupational  areas  in  which  to  invest  training 
monies  to  achieve  the  long-term  goals  and  objectives  of  th^PIC. 

6.     Employment  Opinions  and  Attitudes 

Using  Title  VII  employment  generating  services  monies, 
your  PIC  can  support  employer  surveys  to.  solicit  local  firm 
opinions  or  attitudes  on  the  local  employntent  and  training* 
system  *in  arfeas  such  as: 

•  awareness  of  existing  CEXA  services 

•  their "current  use  of  those  Services 

•  their  interest  in  obtaining  further 

information  on  the  availability  of  *  ' 

*        CETA  employment  and  training  services 

•  quality  and  effectiveness  of  CETA 
services 

•  ,    quality  and  effectiveness  of §  services  *-  v 

received, in  the  past,  provided  by  other  4 
employment  and  training  agencies,  in- 
cluding public  vocational  education 
agencies  and  private  graining  schools.^ 

This  information  can  then  be  utilized  to  provide  recommen- 
dations for  improving  the  effectiveness  of  local  CETA  employment 
iand  training  programs  and  to  develop  an  information  base"* on 
Ipcal  programs  of  "demonstrated  effectiveness" -for  use. in  future 
PIC  fuftdina. 

As  the  preceding  discussion  reveals,  there  is  a 
wide  variety  of  purposes  which  can  be  achieved  through 
employer  surveys.    *A  single  survey  can  be  designed  to  achieve 
several  of  these  purposes.     However,  some  ofthe  purposes  for 
local  employer  surveys  are  i'n  opposition  t(^e  anpther.  Thus, 
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it  is  highly  unlifcrfy  that  your  PIC  would  be*  able  to  devise  one 
employer  survey  which  would  accomplish  the  full  spectrum  of  em- 
ployer survey  purposes,  and  answer  all  the  questions  you  might 
have  regarding  the  local  labor  market. 

Decjgfirig  Whether  tQ  Survey 

While  ii#Drmatioft  gaps  exist  in  available  LMI,  your  PIC 
should  make  the  decision  to  conduct  .an  employer  survey  only  after 
careful  consideration  of  whether  or  not  a  survey  is  justified,  and 
whether  the  expected  outcome  can  actually  be  obtained.  Additional 
data  may  be  needed, ^ut  a  formal  employer  survey  may  not  be  the 
appropriate  data  collection  mechanism.     You  should  take  the  fol- 
lowing preplanning  steps  to  facilitate  your  decision-making  on 
whether  to  survey: 

•  Define  the  information  need.     What  data  are 
needed?    How  will  they  be  used'  in  the  planning 
process?    Will  additional  data,  help  you  to 
make  better  decisions? 

•  Utilize  existing  resources.     S.ep^if  you  can 
get  the  additional  data  you  need  from  tradi- 
tional sources  (e.g.,  the  State  Employment 
Security  Agency  (SESA) ,  and  State  Occupa-- 
tional  Information  Coordinating  Committee 
(SOICC) .   i  >  ' 

•  Assess  th^  survey.     Make  a  careful  assess-* 
ment  of  the  costs  of  the  survey,  correct 
survey  procedures,  reliability  of  data  to 
be  gathered,  and  possible  alternatives  to  , 
the  surveyl  "  t 

•  Make  the  decision.    Reevaluate  your  total 
information  requirements  to  see  if  they 
can  be  reduced,  whether  alternative  data 

*  can  be  used,  and  whether  certain  iiata 

needs  can  bb  dropped  without  adverse 
impact  on  t^ie  planning  process. 

•  Coordinate  with  your  SESA  and  SOICC.  Once 

.    -  you  decide  %o  do  a  fbnnal  employer  survey,  t, 
'  contact  you*  SESA  and  SOICC  to  take'  ad- 
vantage of  any  related  activity  and  to 
f    *   secure  their,  assistance  in  your  project. 

•  Solicit  additional  assistance.     Seek  aid  from 
your  Prime  Sponsor,  local  universities,  and 

'  others  who  m^ght  be  able  to  help  you  in 
implementing  \ your  survey. 
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Implementing  an  Employer  Survey  n  * 

If  you  decide  that  y|iur  PIC  needs  to  do  an  employer  survey (s) 
then  you  must  take  great  care 'to  ensure  that  thfe *  survey (s)  is (are) 
designed  in  a  careful  and  rigorous  manner  and  administered 
efficiently.    To  ensure  this  end,  you  should?  f% 

•  Think  out  the  goal  for  th^  survey  carefully 
and  completely,  in.  advance  of  its  design  » 

•  Dgjpiop  thd  survey  instrument  based  upon 
cSpcepts  that  are  measurable  and  meaningful 

•  Prepare  the  sample  design  using  a  compre- 
hensive sample  frame  (all  industries  within  the 
scope  of  survey)  and  taking  the  objectives 

of  the  survey  and  the  leyel  of  resources 
available^  into  account  $ 

•  Establish  effective  int^rnal^jnanagement 
procedures  to  guarantee  successful 
administration  of  the  survey  *  "  • 

•  Collect  data  and  tabulate  results. 

Some  additional  summary  information  on  the  foregoing  pro- 
cedures is  provided  in  the  following  p^ges^ 

*  I.     Developing  Your  Survey  Goals 

The*  goals  an^objectiVes  which  you  set  for  your  local  survey 
effort  will  substantially  shape  othfer  major  components  of  the 
survey,  particularly  the  sample  design,,  the  develppmerit  of- the 
sur.vey  instrument ,  and  the  selection  of  .data  collection  tech- 
niqyes.  . 

Two  general  types  6f  goals  may  be  established  -for  local 
employer  surveys:  ^ 

'  Analytical  goal  -  Used  when  the  primary 
purpose  of  the  loqal  survey  is  td  develop 
statistically  reliable  data  for  research, 
planning  and/or  evaluation  purposes. 

.  Operational  gorfl  -  Used  when  the  primary  purpose  * 
,        of  the  Survey  is  to  assist  in  thfe  improvement 
*         of  the  ongoing  administration  of  the  local, 
employment  and  training  delivery  system.  ' 
This  would  include  efforts  to  improve  the  ✓ 
job  development  and  placement  performance 
,  '       of  the  system  or  to'  .enhance  the'  marketing 
^         +  of  CETA  services  to  local  business  estab- 
>'  lishments. 
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*    Frequently,  there  is  a  desire  to  accomplish  both  types  of 

goals  with 'a  single  survey.    Unfortunately,  designing  a  survey5" 

that  is  hath  analytically  and  operationally  useftil  can  bfe  quite 

difificult  ^Oftentimes,  the  gpals  of  analytical  ^and  operational 
,  i.*-  *  '  *  *  * 

nsefuV^^s^s  will  be  in  conflict  with  one  another.     For,,  example, 

a  survey  designed  to  produce  information, to  support  planning, 


research  an4  fevalu^ion  efforts  wouldyfcjenerally  be  bashed  upon 
pVpceaB^4Sivolving  the  random  selection  of-'-firn^  to  partici- 
pate in  the  survey.    An  effort  to  (market  CETA  services,  such* 
t 33  OJT,  however,  would^ ideally  b£>  targeted  at  a  specific  set 
V^Y^jf  ^  f^ns  identified  as  most  likely  to  be  in  need  of  such  ser- 
1       vices  ,or  amenable  to  the  *  provision/of .  such  ^rvices. 

^         Your  PIC's  critical  first  ^tep  in  the  development  of  a 
local  employer  survey  is  to. define \explicitly  its  set  of  over^ 
4    -al-l',  sturvey  goals..    In  doing,  this,  ^xai&ine  all  goals  tha£  you 
establish  to  ensure  that  they  are  not  inVconflict  .with  orte  y 
another*  v  ,1  „ 


~2\     Developing  Your  Survey  Concepts 


Devplopiilg  your  "survey  conckpts^is  the  'second  step 


toward  implementtng- your  FIC ' s  employer  Survey^    The  concepts 
which  you  emplby  irf  your  employer  survey  should  be  developed 
so  that  they  will  be  both  meaningful  and  measurable? 

Meaningful:-    To  be  meaningful  t%  both. the^ data  analyst^as 
well  as  the  respondent,  a  survey  concept  must  ^e  defined  in  a 
readily  understandable,  yfet  rigorous  and  explicit  -Planner,  tor. 
wexajtople,  If,  a  survey  quest ionnair-e  is  developed  to  ask  private 
sector  empioyers^hat  their  "manpo^fer  needs"  ate,  th€»  set  of 
responses.fltould  likely  differ  quite.,  sharply 4 from  those  that 
woml£  be  provided  on  the  basis  of  a  questionnaire  that  asked 
employers  to  list  the  "number  of  Jbb  openings  that  were  imme- 
di^eX^PSSifaAlable  for' occupancy  by  workers  from  outside  this 
given  point  in  time."  ' 
The  "manpower  needs"  concept^  as. stated  above,  is  not  suffi-. 
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to  .be  clear  to  many'  survey  respondents  exactly  what  t^pe  of 
information  is- be^ng  ^sought .    As  a  result,  tfteset  of  responses 
A    provided  by  cooperating  'establishments  will  be  based  on  their  ' 
r  pwnrte^ipns  of  manpower  needs.     These  notions  may  vary  con- 
siderably among  respondents.  * 

teasurable;    Your  employer  survey  concepts  must  alsa  be 
ned  so  that  they  are  readily  measurable,  i.e,,  the  survey 
confcepts.  need  -to  be  defined  in  a  manner  -that  the-  respondents 
can  both  easily  understand,  and  retrieve  or  secui^'  the  data 
ttiey  frequest.     In  the  case  of  the  data  requested,  individuals 
in  the  establishments  contacted   (e.g.,  personnel  officers)  must" 
•possess  the  information  required  to  respond  accurately' to  the 
questionnaire."  * 

Whil^  survey  concepts  can  generally  be  defined  in  a  readily 
understandable  fashion,  the  problem  of  inadequate  information 
upon  which  to  base  a  reasonably  accurate,  response  may  still 
exist. J  It  is  typically  the  case: that  information  related  to 
recent  or  current  fifm  behavior  is  more  likely  to 'exist  within 
the  responding  unit  than  information  a£qut  a  firm1  ^intentions 
in  the  future.  & 

For  example,  Ajp.  establishment  may  or  may  not  possess  inter- 
nall^ata^ that  would' allow  it  to  readily  supply  adcurate  infor- 
mation on  ita> pa strand • current  levels  of  employment.     Yet,  with 
few  exceptions,  information  with  resect,  to.  firms'  anticipated 
employment' levels  at  some*5uturd  date  (1-2  years)  does  not  # 
readily  exists  ^ 

You  gan  enhance  you**  surveys  considerably  by  devising  local 
survey  concepts  tfcatare  comparable  to  dfc  compatible  with  exist-  ' 
in$  data  sources.     Therefore,  to  the  extent  possible,  you  should 
attempt  to  match  yoy^ concepts  to  the  concepts  and  clarification 
systSm  used  in  t}ie  state.     This  will  allow  you  to  do  comparative 
analyses  of  items  such  as  job  openings  by  occupation  and  unem- 
ployment by  area.     Finally,  as  an  essential  part  of  developing  ^ 
your  survey  concepts,  you  should  determine  how  th& data* obtained 
will  be  used  by  the^ PIC  members  and  staff. 
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3 Preparing  'Your  Sampling  Design 

Preparing  your  sampling  design  consists  of:  defining 
your  survey's  scope;  determining  -your  sample  frame;  deciding 
upon  sample  size;  and  then  selecting  your, sample. 

Survey'  Scope :    Your  survey  scope*  refers  to  the  population 
/  *   of  business  establishments  about  whom  you  want  to  know  some- 
thing.    Depending  upon  the  goal  o.f  your  survey,  y^mr  scope  will' ' 
be  either  broadly-  or  narrowly-defined. 

-For  example,  if  your  Pic  wants  to  secure  data'  regarding 
the,  occupational  distribution  of  new  hires  taking  place  within* 
•  the.  local  labor  markets,  ti^ .scope  of  your  survey  would  be 
broa'd  and  might  include  all  nonagricultural  business  establish- 
ments.    If,  pnthe  other  hand,  your  PIC  has  already  selected  a 
set  o.f  specific  industry-occupational  job  clusters  that  it 
believes  to  be  suitably  for  training,   but  only  wants  to  Verify 
their  suitability,  your  scope  woulf  be"  more  narrow  or  focused. 
"  •       You  could  dfsign/your  focused  survey  to  producer  wide  variety 
^f^n  format  ion  about  the  characteristics  (wages,  internal  . 
promotional,  opportunities),  hiring  requirements,  training  re- ' 
quirements)   within  those  (specific  clusters  which  you  have 
selected.  ' 

In  general,  brdfea"  scope  «e$aployer  surveys  are  more  difficult 
to  conduct  than  more  formed  surveys.  *   :    .  • 

Sampling  frame:     After  determining"  the  scope  of  your  em- 
•  ployer  survey,  you -next -determine  a  sampling  frame.,,  At  a  mini- 
mum, this  sample  frame  should  be  capable  of  identifying  the  . 
name,  address  and'  industrial  classification  pf  the  entire' set— * 
of  establishments  that  are  within  the  survey's  scope.  most 
comprehensive  source  from, which  you  can  derive  such  establish- 
ment^ information  is/ the  unemployment  insurance  tax^gports  ' 
(Form  ES-202)  provided  to  your  SESA- by  all  employers,  within  , 
the  state  that  are  cWred  by  the' federal  and  state  Unemployment' 
insurance  laws.  * 

The  public  availability  of  sampling  frame  data  from  the" 
ES-202,  however,  varies ,from  state  to  state.   ' Some , states 
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consider  such  "information '  t,o  be  <*»nfidential  in  nature  and  will 
not  identify  Specific  establishmenf^when  providing  any  type  of  - 

.information  to  individuals  outside- of  the^SESA  system.  While 
other  states  are  unwilling  to  provide  specific-employment  data 
for- individual  establishments,  they  will  provide  PIC  staff  .with 
establishment  listings  that  identify  firms'  names,  addresses, 
industry  classification,,  and -employment/size  classes.  There-J^ 
fore',  as  a  first  step  in  procuring  a  sampling  frame,  you  should 
contact  the  Research  and' Statistics  section  of  the  SESA  to  dis- 
cuss the  possibilities"  of  securing  the  ES-202  sample  frame  listings 
if  establishment  address  listings  from  the  ES-202  file  are 

.not  available,  you  should  consider  using  exo.stj.ng  industrial  direc- 
tories-to  determine  the  sampling  fi^me^It  is  important  to  note, 
hov/ever;  that  the  use  of  such  d/Lrector ies  may  require  a  substan- 
tial alteration  in  the  scope  of  data  collection  efforts.  Only 

v  rarely  will  such  directories  include  "all  of  the  business  establish- 
ments} with/CrT*  selected  subset  of  industries  within  an  area.  . 

Consequently,  if  you 'are  forced  to  draw  your  sampling  frame 
from  industrial  directories,  both  the  scope  of  your  study  and  the 
gerieralizability  of'your  -findings  Will  be  necessarily  reduced. 
Because  the  directories  are  not  comprehensive,  you  cannot  be 
confident  ttiat  yoUr  sample  will  accurately  represent  all  local 
firms  in  the > industry-occupation  under  examination*  -As  a'result, 
it  Would  be  extremely  difficult  to  develop  statistically  reliable  ^ 
industry-wide  estimations  for  variables  such  as  new  hires  or  job* 

v openings .  ,  r        ,     .  - 

'  '  •  Local  industrial  directories,  therefore,  are  best* suited  for 
>  conducting  surveys  designed  to  provide  insights  into  the  activi-, 

ties  of  a  selected  set  of  individual  firms.  This  data- may  poten- 
•  tially  prove  quite  useful  for  your  PIC  .in  developing  profile?  of 

specific  local  business  firms,   including  their  internal  labor 

markets,  hiring  policies  and  practices",  wage  policies,  and  train- 

ihg  practices. 

Sample  Size;  /After  procuring  your  sampling  frame,  the  re- 
'  maining  steps  are  /determining  the  overall  sample  size  for  the 
survey  a,nd  then  developing  methods  for.  selecting  establishments 
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for  inclusion  within  the  sample.     If  your  PIC  intends  to  use  the  * 
employer  survey  to  produce  statistically  reliable  estimates  of  1 
labor  market  variables,  two  general  factors  must  be  considered 
in  determining  the  appropriate  sample  size.  A_ 

First,  consideration  must  be  given  to  the  desired  degree 
of  confidence  that  you  would  like  to  have  about  the  estimates 
that  the  survey  .will  produce.    Secondly,  you  must  consider  the 
amount  of  resources  that  you  are  willing  to  devote  "'to  the  sur- 
vey's operation. 

In  a  probability  based  survey,  it  is  generally  true  ttf 
creasing 'the  size  of  the  sample  will  lead  to  decreasing  sampling 
error  associated  with  estimate^produced  by  the  survey.  'Sampling 
error  can  be  thought  of  as  aj  measure  of  the  potential  degree  to  * 
which  the  mean  6f  a  given  sample  may  fail  to  truly  reflect  the 
mean  of  the  entire  population'. 

Sampling  error*  results  frbm  the  fact  that  observations  are 
made  on  only  some  fraction^of  the  population  of  establishments. 
Sampling  error  is  also  influenced  by "variations  or  differences  in 
the  characteristics  of  elements  of  the  population.    For  example, 
if  your  PIC  wished  td*measure/new  hires  by  qpcupation  within 
local  business  establishments,  the  sample  size  required  ta  do  so 
tould  depend  in  part  on  the  degree  "to  which  and  how  such  new 
hires 'occurred  among  fii?ms.\ 

If  all  firms  hired  the  same  proportion  of  new  workers  into 
similar  occupations,  required  Sample  sizes  would  be  relatively 
small,  even  if  a  high  degree""  of  confidence  is  de'sired.     I£,  how- 
ever, local  firms  tend  to  hire  sharply  different  proportions  of 
workers  ii\p  diverse  sets  of  occupations,  then  the  skmple  sKze 
required  to  product?* estimates  with  a  giVen  level  of  confidence, 
would  be  felatitfply  large.    Thus,  on  the  basis* of  purely  statis- 
tical criteria,  the  sample  size  you  emplojuwill  be  determined  by 
/the  amount*  of,  sailing  error  that  your  PIC  l^^lling  to  tolerate 
and  by  the  relative  homogd^lity  of  the  population  to  be  studied. 
A  second  cop#tderation  which  you  must  also  take  into  account 
determining  a  proportionate  sampl^  size  is  the  level  of  re- 
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sources  that  your  PIC  is  willing  to  devote  to  the  data  collection 
effort.     For  example,  using  the  statistical  approach  discussed 
above^  it  may  Jpe  found  that  a  sample  of  1500  establishments  isT 
required  to  produce  an  estimate  with  the  desired  degree  of 
statistical  confidence. 

Your  PIC  may  b£  willing  to  spend  $30,000  on  its  survey. 
Contacts  with  CETA  Prime  Sponsors,,  the  SOICG,  and  the  SESA  may 
reveal,  however,  that  the  likely  cost  of  convicting  ;such  a  survey 
will  b£  $30.00  per  sample  unit.     Tfts^total  estimated  cost  of       •  4 
sampling  these  1500  units  would  thus  be  $45,000.  Consequently 
in  order  to  stay  within  a  predetermined*  budget,  ^ou^would  have 
to  reduce  your  sample  size  to  1000  units'.* 

The  effect  of  this  sample  reduction  will  be  tcr  raise^the 
expected  sampling^^rror  associated  with  the  estimate.     Developing  .x 
sample  sizes  based  an  a  realistic  appraisal-  of  per  unit  costs' 
and  the  overall    survey  budget  is  an  appropriate  method  for 
determining  the  sample  size*     The  primary  thing  that  you  should'  ~~ 
remember  is  that  when  you  have  to  adjust  sample  size  downward  to" 
fit  your  survey  budget,  your  degree*  of  confidence  must  also  be 
reduced  somewhat.  .  *  * 

Finally,   if  your  PIC  .has  chosen  tfa  use  employer  surveys  not 
to  develop  statistically  reliable .estimates  byt'for  informational 
purposes,  then  the  sample  size  issue  becomes  primarily  a  matter 
related  to  the  av^ij^fbility  of  overall  resources.     Again,  con-  *T, 
sideration  should  be  given  to  linit  costs  and  >the  size  of  the  * 
overall  survey  biidget. 

Sample  Selectlpn;    The  ^manner  in  which  you  select  your  sample 
of  establishments  will  depend  mpon  the  purpose,  of  your  survey, 
i.e.,  whether  the  local"  survey  is,  designed  to  produce  statistically 
reliable  estimates  with  'a  -given,  degree'  of  confidence  or  whether  it 
isfdesigned  to  prpduce  information^  on '  a  specific  subset  .of  local 
firms.       *  * 

If  the  survey's  overall  goal  is  to  produce  statistically 
reliable  data,  then  a. random  sample  selection  process  with  a 


known  probability  of  selection  is  required.    Random  selection 
simply  means  that,  within  a  given  population,  each  unit  (economic 
establishment)  hag  the  same  probabili€y  of  being  included  within 
the  sample.    It  is  equally- important  to  know  the  probability  of 
a  given  unit  being  selected  (the  sample  ratio) ,  since  the  inverse 
of  tfcis  particular  probability  will  be  used  to  make  inferences 
about  the  overall  'population  within  the  scope  of  the  survey. 
Forjexample*  if  a  sample  is  designed  to  select  one  of  every  three 
units  in  the  population,   its  sampling  ratio  is  1:3.     The  inverse 
of  this  ratio  (3)  becomes*  the  weight  upon  which  estimates  fo^r 
the  entire  population  will  be  based. 

If  the  overall  goal  of  your  local  employer  survey  is  to. 
proiJuce  information  on  the  hiring,  training, ,  and  wage  policies 
of  local  firms,  or  to  develop  information  for  use  in  Prime  sponsor 
program  operations,  such  as  job  placement then  random  selection 
of  units  for  inclusion    within  the  survey  is  'probably  inefficient. 
A  more  efficient  alternative  would  be  to  target  the  local  data 
collection  effects  upon  a  particular  set  of  firms  within  the  ' 
local  labor  area. 

Using  existing  sources,  your  PIC  can  select,  firms  for 
inclusion  in  the  survey.     In  this  way,  the  sample: can  be  targeted 
at  specific  establishments' about  whom  your  PIC  would  like  to 
learn  more.     It  is  important  to  remember,  however,  that  selec- 
tion of  units  in  this  way  *  precludes  the, development  of  statis-  ' 
tically  reliable  estimates  for  the  universe  of  establishments, 
in  conclusion,  given  the  existing  gaps  in  LMI,   it  is  anticipated 
that  the  nonrandom  approach  to  establishment  sample  selection 
would  be  the  one  which  you  would  employ  most  frequently  ;Ln 
meeting  your  PIC  data  needs." 

4 .     Establishing  Your  Internal  Management  System 

|Once  you  have  prepared  your  saihple  design,  your  next  respon- 
sibility is  to  develop  effective,  techniques  for  managing  the  da'ta 
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collection  process  internally  in  order  to  allow  for  the  produc- 
tion of  timely  and  accurate  information.    Two  fundamental 
components,  for "your  internal  management  system  should  be  a 
control  file  and  an  internal  schedule  flow.  - 

•  Control  File:     The  basic  tool  used  to  manage  a  local 
employer  survey  is  a  control  file,.    A  control  file  is  composed 
of  a  list  of  sample  establishments  including  the  name,  address,. 
SIC  industry  classification'  code,  employment  level,  if 
available,  and  a  unique  identification  number.     The  control 
file  is  updated  regularly         used  to  monitpr  changes  in  the 
response  status  of  individual  units  included  within  the  safmple. 

The  control  file  is  used  to  monitor  the  overall  rate  of 
response  to  the  survey.    Based  uporv  inf^sujation  contained 
withirv"  the  control  file,  follow-up  data  collect io.j  efforts 
can  be  focused  upon  specific  areas  where  response  rates  are 
relatively  weak.     Control  files  can  also  be. used  to  record 
information  on  follow-up  activities.     Space  should  fcfe  pro- 
vided on  such  files  .tofr6co2fd  the  type  (mail,  visit/  telephone) 
of  follow-up  und^rt^en  and  the  date  that  such  follow-up 
contacts  were  made. 

Schedule  Flow;     In  addition  to  employing  control  files 
to  manage  the  flow  of  information  between ^your  PIC  and  the 
sample  of  local  establishments,  an  internal  schedule  flow 
also  needs  to  be* developed.     This,  schedule  flow  should  define 
a  process  consisting,  at  a  minimum,  of  the  following: 

•  A  schedule  checking  in  and  acknowledging 
receipt  of  a  response  .on  the  control  file  ^  

•  Editing  and  screening  procedures,  involving 
a  review  of  the  information  provided  by 

•  respondents  to  insure  internal  consistency. 
These  procedures  include  comparisons  of  con-  * 

'      '   trol  file  information  with  response  information 

•  Follow-up  contacts  with  establishments  to  • 
Obtain  additional  information  or  to  correct 
inconsistencies  in  the  information  provided  * 
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•  Updates  of  the  response  status  of  establish- 
ments on  the  control  files,  (e.g.,  usable, 
non-usable)       *  . 

•  File  schedules  for  use  in  producing  data 
tabulations  at  the  completion  of  the  data* 
collection  effort. 

In  seating  up" your  internal  management  system,  you  should 
consult  with  the  SESA  research  staff  to  select  effective  \ 
methods  for  organizing  and  operation.     In  addition,  you  mityht 
want  to  refer  to  your  BLS  publication,     PES  Survey  Operations 
Manual,  which  provides  a  detailed  discussion  on  techniques 
for  organizing  a  large  (fata  collection  effort,  ^ 

5.    Collecting  Ytmr  Data  n 


Your  data  collection  process  w^JX  typically  involve  three  • 

#> 

parts:     initiation,  data  collection,  and  follow-up. 

Initiation;     The  process  of  initiation  consists  of  a 
s6t  of  efforts  that  are  designee^  to  enlist  emplpyer  estab- 
lishments in  the  survey  effort,  and  to  identify  the-  individual 
or  individuals  within  the  establishment  who  possess  or  have 
access  to  the  desired  information. 

Initiation  is  not  intended  to  directly  generate  informa- 
tion' on  .  establishment  behavior?    Instead,  its  purpose  is  to 
identify  a  specific  contact'  persqn  within  the  establishment, 
to  explain  the  survey* s  purpose,  describe  the  features  of 
the  questionnaire*  that  will  be  administered  as  part  of  the 
sjjjvey,  and  solicit  the  qooperation  t>f  establishments. 
Generally,  initiation, of  this  type  is  carried  out  over  the 
telephone  prior  to  the  efctual-  data  collection  dffort,  or 
thrdugh  a  well-designed,  presurvey  letter  followed  by  a 
telephone  cal 1 .  t 

Data  Collect ion;     Aftet  initiation  has  been  completed, 
the  actual  process  of  data  collection  can  be  undertaken.  One 
of  three  approaches  (mail,  telephone,  or  personal  visit)  can 
be  employed  to  collect; the  necessary  information  from  employers. 
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Depending  upon  the  types  of  questions  asked  and  the  employ- 
ment size  distribution  of  the  sample  of  establishments,  any 
one  or  a  combination  of  approaches  may  be  appropriate. 

If  the  questionnaire  is  brief  and  it  is  believed  tha^ 
the  information  is  readily  accessible  tcp. respondents,  then 
data  collection  directly  over .  tjie  'telephone  may  be  most  appro- 
priate.    If  the  questionnaire  is  more  complex,  requesting  ,  -  * 
information  on  a  variety  of  topics,  personal  visits  may  be 
required.     Due  tothe  lower  degree  of  organizational  com- 
plexity, vithin  the  smaller  establishments,  phone  collection 
may  prove  to  be  the  most  effective  strategy.     In  conclusion, 
although  mail  questionnaires  h^ve  often"  served  as  a  standard 
method  for  conducting  establishment  purveys,  telephone  and 
personal  yis4ts,  at  times,  can  serve  as  more  effective  approache 
for  producing  the  desired  information. 

Follow- Up:     Typically,  you  can,  anticipate  that  the 
initial  attempt  to  obtain  information  from  employers  will 
not  produce  an  adequate  overall  rate  of  response.     In  fact, 
you  should  expect  that  the^bulk  of  the  response  genera-ted  by 
the* survey  effort  will  occur  only ^af ter . at  least  one  follow-up 
contact  has  been  made.     Therefore,  an  important  part  *pf  your 
data  collection  process  rjust  be  a  .detailed  plan  for  follow-up. 

,  Experience  has  shown  that  follow-up  contacts  made  through 
the  mail  are  likely  to  prove  relatively  ineffective.    As  a 
result,  your  follow-up  efforts  should  probably  consist  primarily 
of  telephone  reminders,  direct  collection  of  information  over 
the  phone,,  or  field  visits. 

6.    .Tabulating  thfe  Results  of  the  Survey 

You  should  start  thinking  through  the  types  pf, tabula- 
tions desired  simultaneously  with  tfie  development  of  your 
overall  survey  goals;    A  useful  step  in  this  process,  xs 
developing  hypothetical  table  formats  at  the  initial  planning 
stages  of  the  survey.     The  preparation  -of  slich  tables  will 
contribute-  to  the  clarification  of  a  number  of  issues  related 
to  thfe  overall  design  of  the  survey  and  Will  aid  in  avoiding 
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number  of  problems  associated  with  producing  desirfefl  tabula- 
tions after  the  qata  collection  effort  is  completed. 

"There  are  three  primary  types  of  data  tabulation  which 
might  be  possible  based  upon  the  nature  of  your  questionnaire, 
representatives  of  your  sample,  comprehensiveness,  and  the 
reliability  of  ypur  resyises .     These  tabulations  are: 
probability  estiifates;  data  summation  and  cross-tabulation;  ^ 
and  individual  J.ndustry  or  firm  profiles 

As  noted  earlier,  structuring  a  sample  of  sufficient 
size  and  representativeness  to  ensure  strongly  reliable  , 
probability  estimates  (I.e.,  what  evidence  suggests  will 
happen  in  the  future)   for  factors  such  a^  new  hire  rates  and 
turnover  roles,  might  be  difficult.     However,  if  your  PIC  has* 
been  able  to  overcome  this 'problem,  at  this  point,  you  will  be 
applying  var^ouSj  statistical  techniques  to  establish  probability 
based  upon  jrour  results.     These  statistical  treatments  require 
sophistication  of  the  user  and  will  probably  require  a  computer 

system's  processing  capability.  \ 
'  -  -        #  >■>    _  • 

Data  summation  and  cross-tabulation  4a  a  much  less  com- 

ple'x  treatment  process.     This  type  of  data  treatment  simply 

involves  the  preparation  of  a  .number        tables  designed  to 

summarize  and/or- to  compare  the  data  contained  on  responses 

received  from*  cooperating  sample  establishments.     For  example, 

if  700  local  employers  rej/orted  new  hires. data  by , occupation,  • 

then  the  tables  produced  would  be  developed  by  summing  the 

*•* 

responses  by  occupation  across  all  700  usable  resources. 
•Other  tables  could  be  produced  to  break  out  and  compare  new 
hires  data  by  industry  and  occupation.*    Such  summary  tables 
a*e  relatively  easy  to  produce  and  can  be  completed  manually. 
Although  tables  of  this  type  do  not  provide  information  about  . 
those  employers  that  fail  to  respond  to  the  survey,  they 
may  be  capable* of  providing  substantial  insights  into  the 
hiring  behavior  of  a  substantial,  component  of  the  local  labor 
market. 
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Finally,  your  PIC  could  choose  to  tabulate  your  data 
results  to  develop  individual  profiles  on  the  hiring,  training, 
and  promotion  practice  of  local  employers.    ,Such  information 
organized  by  occupation  on  an  individual  employer  basis  could 
serve  a  wide  variety  of  counseling,  planning,  and  job  develop- 
ment  purposes  within  the  PIC. 


;  CONCLUSION 

In  conclusion,  a  significant  amount'  of  space  has  been 
devoted  to  this  step.     This  is  because  labot  market  analysis 
is  the  cornerstone  for  effective  PIC  programming.     The  more 
that  you  can  do  to  ensure  that' your  anal^lks^  is  accurate 
and  meets  your  decision-m^jng  ne£ds,  the  greater  the  prob- 
ability that  yob  will  design  a  progtam  which  is  responsive 
to  local  needs  ^nd  thus  capable  of  success. 


The  •  inf  ormafclon^jpresented  Under  this 
heading  has  been  condensed  and  abstracted 
"£f  om "a"  recent  Department  of  I^bor  publica- 
tion, Jobs  in  the  Private  Sector:     Use  of 
Labor  Market  Information  (Washihgton,  D.C 
U.S.  Government  Printing  Office,  October 
1980)  .  '  . 
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Step  2 


Analyze  Local  Population  Groups  and  Service  Needs 


The  purpose  of  this  step  is  to  identify  those  population  -groups 
within  your  local  .community  who  are  in  need  of  employment  and 
training  services  and  to  document  the  relative  extent  of  th,eir 
need.     The  step  involves  pinpointing,  as  prepisely  as 
possible,  ,the  exact  impediments  and  obstacles  to  job  success 
and  un subsidized  employment  among  these  groups. 


•  Prime  Sponsor  (PS)  Comprehensive  Employment  and  Training 
Plan  (CETP)  :     application,  for  financial  assistance  to  DO: 


 . .   DOL. 

Consists  of  2  parts  -  Master  Plan  and  the  j(Uinual  Plan, 

Master  Plan:     long-term  agreement  between  DOL  and  prime 
Sponsors.     DescriWfc  PS 1  administrative  and  programmatic 
arrangements  of  PS.  *  Contains  demogfcaphic  characteristic^ 
which  a,  PIC  can  review  to  assess  population  groups  and 
their  service  nee$3s. 


ERLC 


tor 


Knowledoe/CaoabilUjes  Rating 


Review  the  Information  provided  above.  Rate  your  flic's  knowledge  and 
capabilities  with  reference  to  this  step  on  the  scales  bel^pw. 


Knowledge 


•  Capabilities 


L 


Excel- 
lent 


Excel- 
lent 


Above* 
Avg. 


J. 


Above 
Avg. 


Aver-     I  jBelow 
age  i&/^vg, 


Poor 


Aver- 
age 


Based  upon  your  ^ratings,  decide  whether  you  need  to 
information  on  this  step  which  follows  immediately  o 
next  step. 


contept 
to  the 
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-  PME  PROCESS  ACTION  PLANNING. PAGE 

t^TE^^TrttT^ha^^^mporta^  remamber) 


*  J 


QUESTIONS  (Items  on  which  you  are  unclear  or  want  to  get  actional  - 
information) 

r  ^  * 

-I 


ACTIONS .(Specific  steps^that  you  intend  to  take* to  implement  your 
*  PlC's  PME  process  in  this  .atrea) 


Steps 


Resources/Persons 


Time  Frame 


4 


er|cv 
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*  '  INTRODUCTION 

 * — : 

Your  PJC's  responsibility  in  acco.mplishing  this  step  is 
neither  as  substantial  nor  as  primary  as-*  in  tile"  preceding  step 
of  local  labor  market  analysis.    Still,  it  is.  essential  that  . 
Y  your 'PIC  be  able  to  secure  accuralfe  data  on' the  most  pressing 
grolip  and  services  needs  within  the  local  community.  Quality 
•,data  in  these  areas  will  enable  you  to'  design  and  .develop  .  % 
effective  employment  and  training  pro jefi^^targigted  directly 
toward  the  needs -of  the  *un-  and  underemployed"  segments  of 
your  population* '  -     .       \   ■  ^ 

Prime  Sponsor's  Responsibility  •      •  * 

The  major. burden  for  collecting. and  analyzing  population  and 
service  needs  has  historically  fallen  upon  the  Prime  Sponsors   The  CETA 
•    "Regulations  require  that  the  Master  Plan  portion  of  the  Prime 
•  Sponsor's  CETP  furnish  "a  detailed  description  of  the  demo-     -  * 
graphic  characteristics  of  the  population. . and  eligible  par- 
ticipants. .. "  '    •  (  * 

To  satisfy  jthis  requirement  minimally,  as  it  is  interpreted  • 
in  the  Department  of  Labor's  CETA  Forms'  Preparation  Handbook, 
the  Prime  Sponsor  "must  identify  the  portion  of  th£  local 
'resident  population  which",  by  virtue' of  employment  status  and 
income,  is^tentially  eligible  for  CETA  programs;"  by  com-     ^  # 
pleting  the2  chart  on  the  following  page  "using  the  besff  s 
available  estimates  or  data."    The  population  groups  broken  . 
jyjt  on  -.the  chart,  are  referred  to  by  the  Department  of  Labor    ,  ^ 
,  'as  ^significant  segments.-"        '  ' 

*       In  addition,  in  the  Forms  Preparation  Handbook  the  Prime 
Sponsor* is  ".. .encouraged- to  gather  more  detailed  information, 
"where  available,'  as  to  the  demographic  characteristics  of  the 
'  eligible  population,  including  .characteristics  data  cross*r 
tabulated  by  age,  race,  and  sex...";  and,  "to  study  locally... 
'    factors  including  employment  status,  household  status, • level  of 
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Total ,  For  PS  Area: 


DEMOGRAPHIC  GROUPS 


Sex 


Age 


Race/ 

Ethnic 

-Croup 


Male 


Female 


19  .under 


20  -  21 


55  and  over 


fchite  (not 
Hispanic) 


Black  (not 
Hispanic) 


Hispanic 


American  Indian 
Alaskan  Native  * 


Asian  or  Pacific 

Islander 
—  * 


Estimated 
number  of 
persons 
potentially' 
Geoeral         eligible*  for 
Population        CETA  Programs 


) 


■  \ 
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employability,  -handicap,  veterar/  status?. .as  well  as  other 
t factors  identified  by  the"  Prime  Sponsor:"  ^  .  - 

A.     REVIEWING  THE  MASTER  PLAN  *  * 

Given  the  foregoing  requirements  arid  directions,  your 
PIC's  initial^reference  source  for  data  regarding  population 
groupa^andtheir  service  needs#should  be  ,the  Prime  Sponsor's 
Master  Plan.     You  #should  revitew. those  parts  of  the  Elan  which 
adcTress  population  and  m^tljpds  for  determining  priorities  for  * 
service  amrfng  the  CETA-eliqible  population  to  determine  < 
exactly  how  local^population  groups  *and  their  needs  have-  been 
analyzed  and  what  the  specific  findings  h^ve  been.- 
*      Your'  purpose  in  reviewing  ^this  data  shonid  be  to  ascertain 
whether  it  is  valid, ' reliable,  and  detailed  sufficiently  to 
allow  you  to  use  it  in  making  decisions  on  which  target  groups 
he  PIC  should  serve  and  what  programs  and  services  to  provide. 
Criteria  which  you  can  employ  in  making  this  review  include: 

timelines  ,  / 

comprehensiveness  . 
analytical^  insight  .<* 
particularization, 

problematic  insight.  \ 


Crj 


Timeliness  * 

Obviously,"  one  of  the  necessary  characteristics  of 
useful  employment  and  training  data  is  -that  it  be  as  current 
as  posslbJ#.     The  CETA  regulations  require  that  the  PS 
indicate  the  sotftce(s)  of  data  usecl  and  the  pe'riod  of  tiraf? 
represented.     In  your  review  of  the  data  contained  in  the 
Prime  Sponsor1 s  Master  Plan,  you  want  to  check  both  the 
sources  and  their  ti^rle  periods  to  make  sure  they  are  reflec- 
tive of  present  conditions  within  yout  geo'g^phic  areai 
i  f 

Comprehensiveness  ^   •  -  - 

There  are  a  number  of  sources  which  can.be  consulted 
to  develop  estimates  of -the  population>*4.gible  for  CETA 
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participation.     These  include:  "x-^ 

SESA  data  on  the* number  of  persons  un- 
employed 15  weeks  or  more 

census  ciata  on  the  number  of  persons 
residinq  in  households  having  income 
less  than  OMB  povprty  guidelines 

•        local ,   state,  and, federal  data  on  the 
number  of  persons  receiving  AFDC ,  or 
State  or  local  public  assistance. 

There  are  some  basic  problems  with  all  of  these  sources  of 
data: 


Each  data  base  gives  only  a  partial 
.  picture^pf  the  employment  and  draining 

m  t 


nee<jf 


the  local  community 


#  The  data  bases  for  each  source  serve 
different. purposes,  are  independent 
of  one  another,  and  thus  give  very 

#       different  perspectives  upon  the 

economic  situation  of  locality  * 

#  None  of  the  3a ta -bases  has  been  de- 
signed specifically  to  meet  the  needs 
of  local  employment  and  training 
planners. 

*  m 

Given  these  very  serious  limitations  upon  existing  data 
bases,  you  should  determine  in  your  review  o^ the  comprehen- 
siveness of  the  Master  *£lan  whether: 

•  more  than  one  standard  source  was 
employed  to  analyze  and  define  popula- 
tion groups  and  their  needs 

•  iocally-generated  data  have  been  employed 
to  supplement  existing  (Jdata  and  to  re- 
fine and  complete  the  d^ficnption  of  - 
the  "universe  of  need"  within  the  com- 
munity. 

•  If  the  Master  Plan  has  been  developed  using  several 

#  t 

existing  data  sources  and  introduces  locally-generated  data 
to  compensate  for  gaps  in  that  in  forma tion^then  you  q^n 
assume  that  the  Plan  has  "face"  comprehensiveness  (i.e., 
it  is  adequate  in, its  general  assessment  and' coverage  cf 
local  population  needs) .  •  # 
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Analytical  Insight 

The  ^Significant  Segments'*  Chart  which  the  Prime  Sponsor 
is  required  to  submit  as  part  of  its  Master  Plan -breaks  out 
estimates  of  potentially  eligible  CETA  participants  into  the 
broad  demographic  categories *of  sex,  age,  and  ^race/ethnic 
group.    While  this  information  provides  a  helpful  overview 
of  the  general  situation  of  j>opulation  groups  within  the  com- 
munity^ it  provides  little  or  no  specific  information  which, 
can  be  -useful  in,  within,  or  across  'group  comparison. 

Therefore,  another  part  of  your  review  of  the  Master 
Plan,  'should  be  to  determine  what  steps  have  been  taken  to 
extend  this  basic  analysis  in 'order  to  f acilitateVdecision- 
making.  Two  cotamon  techniques  which  are  employed  j^y  draw 
some  jlnterrela1*Lonships  and  comparisons  between  m  among 
data  variables  are  percentage  relationships  and  capss- 
tabulations. 

^        Percentage  Relations:     Percentage  relations  peww-t 
examination  of  severity  of^  need  within  and  across  population 
groups,  as  expressed  in  relative  numbers.     A  sample  format 
for  comparing  the  conditions  of  local  population  groups  can  be 

created  by  mpdifying^  the  significant  segments  chart  as 
» 

shown* on  the  following  page.  , 

In  addition,  percentage  rel&^pnships  can  be  developed 
.permit  *a  more  in-depth  or  discrete  look  at  particular  ^ub- 


to .permit 
categories 


ERLC 


For  example, 

labor  force  participation  rates  by  age* 
group,  sex,  and  ralbe/jethnicity  ?  ' 


unemployment  rates  by  age  group,  s 
race/ethnicity;  » 


percent  of  the  population  in  sqhool 
and/or  high  school  drop-out  rates 
by  age  group,  sex,  race^ethnicity; 

percent  of  families  ,with  incomes 
below  the  poverty  level  by  ethnicity; 

percent  of  the  pqpulatioi  by  ethnicity 
receiving  welfare  payments  by  type  of 
payment .        *  t 
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General 
Population 

Estimated 
no. of  persons 
Ayntially 
el7P|^e  for 

CETA  programs, 

Percent  of 
CETA  eligible 
Participants 

Total,  For  PS  Area: 

 %  1  

DEMOGRAPHIC  GROUPS  .  - 

Sex 

Male 

Panale 

• 

< 

19  &  under. 

- 

20  -  21 

* 

22  -  44 

45-54 

— r  .  * 

*  *  * 

• 

i 

55  and  over 

• 

Racfe/ 

Ethnic 

Group 

White  (not 
Hispanic  w_ 

Black  (not  , 
Hispanic 

a 

Hispanic 

 k  

Anerican  Indian 
or  Alaskan  Native 

Asian  or  Pacific 
Islander 

m 
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Cross-tabulation :     Cross-tabulation  is  an  extremely 
valuable  technique  for  obtaining  an  even  more  precise  defini- 
tion of  the  interrelationship  of  factors  by  subgroups  of  , 
population  groups.     For  Example*  the  variables  ofsex,  age, 
and  race/ethnioity  can  be  Cross-tabulated,  as  shown  in  the 
{table  below,  to^efmit  clear  comparison  of  need9  within  and 
across  groups.  * 


Male  -  19  &  Under 


White  (not 


Black  (not 
Hispanic! — 


Hispanic 


Amer.  InQian  or 


Asian  or  pacific 
Islander 


General 

Population  -  Total 


Number  Eligible 
for  CETA  


%  CEIA  b 
Eligible 


4 


Crosfe-tabulatiorte  do  not  have  to  be  restricted  to 
analyzing  data  on* the\ "Significant  Segments"  Chart.  They 
can  be  developed  to  pjpt  and  examine  the  interrelationship 
among  rfy  number  of  variables  (race/ethnicity,  unemployment 
level,  average  level  ofi  educational  achievement,  low  income 
level)  which  seem  worthy  of  investigation. 

.   Again,  as  with  comprehensiveness, '  if  the  Master  Plan 
employs  methods  such  as  percentage  relations,  cross- 
tabulations,  ot  oth^r  statistical  techniques  a>  means  of 
gaining  insignt  into  the\specific  needs/  of  the  local 
population,  th.en  your  P id  should  feel  cdnfident  in  using  - 
the  data  contained  therein. ^ 

Part icularizat ion 

Usiflg  the  demographic!  characteristics  of  age,  sex,  race/ 
ethnicity  is  just  one  way  to  assess  the  economic  needs  of  population 
grobpa/io  tfce.  local  community.     Population  subgroups  can  also 
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be  defined  by  using  characteristics  such  as: 

;  •      disabled  and  Vietnam-era  veterans 

/•       public  assistance  recipients 

:  •      older  Workers  (55  years  of  age  or 
older 

l  •      ex-of fenders 

displaced  homemakers 

handicapped  persons 


•       single  parents 


•  individuals  who  lack  credentials  (such 

as  persons  lacking  licenses,  required  .  * 

experience,  etc.)  ,  4 

•  individuals  who  require  basic  and 

remedial  ^ills  development 

•  alcoholics 

•  mxgrant  workers.  .  % 

The  foregoing, *  or  some  other  population  grouping,  may  be  sig- 
nificant segments — or  especially  high  need  gfoups— within  your 
►PIC's  coninunity  or  geographicvarea .  *  Consequently ^ an  important 
part  of  ~your  review  of  the  Mastei;  Plan  is  determining  whether 
" special"  population  groups  have  been  assessed. 
m 

Problematic  Insight 

«  The  last  type  of  data  to  be  looked  for  .in  the  Master 
Plan  is  daWa  which  pinpoints  major  obstacles  or  problems 
(what  or  why)   inhibiting  successful  employment  for  those 
population  groups  (who)'  which  are  f ound , to  be  in  need  or 
CETA-eligible.     These  obstacles  can  be  defined  through  both 
quantitative  and  qualitative  means  such  its  education  d^ta;  ■ 
interviews  with  members  of  participant  groups;  and  contact^ 
with  IogWi  employers. 

Just  a  few  of  the  factors  which  might  be  identified* 

here  are: 


educational  level 


•  transportation 

•  ^language  * 

•  fchild  care  ' 


III-'SO 


\ 


.  •       world  of  work/employability  orientation 
•  •    skill  development. 

In  your.reviev  of  the  Master  Plan,  you  should  look  for  tenta- 
tive explanations  or  assumptions  on  the  underlying  nature  of  / 
participant  group  need*     This  definition  of  need  shapes  the-  y 
programs  to  be  run  and  the  type  of  services  ^provided  through 
the  Prime  Sponsor.     If  the  need  definition  is  well-constructed, 
then  it  should  be  of  value  to  you  in  defining  potential 
solutions  to  be  implemented  through  the  PIC. 


*    B.     REQUESTING  ADDITIONAL  DATA 

*• 

f      It*  is  likely  that,  you  will  be  able  to  obtain  the  popula- 
tion group  and  service  rie6d  data  which  your  PIC  needs  for 
planning  purposes  from  the  Prime  Sponsor's  Master  Plan., 
If,  on  the  other  hand,  you  believe  some  further  analysis 
.of  existing  data  or  securing  of  additional  data  might  be 
beneficial  to  your  PIC's  decision-makina,  then  vou  and  the  Prime 
Sponsor  should  collaborate  to  ensure  that  either  or  both  things  are  done. 

The  actual  mechanics  for  accomplishing  this  extra 
level  of  population  group  and  service  need  analysis  will 
no  doubt  vary  considerably  from  location  to  location.  Your 
' PIC's  major  concern  in  this  step;  however,  should  'be  to 

ensure!' that  you  have  data  of  sufficient  quant^ty--alTlKauality 

.    •    .  <   ^ 

upon  which  to  base  your  PIC's  program, 
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^Employment  and  Training  Administration, 
CETA  Forms  Preparation  HancTflkok  (Wash- 
ington, D.C.:  U/5~._  Department  of  Labour, 
1980). 

Manpower  Administration  (now  Employment 
and  Training  Administration  [ETA]), 
Manpower  Program  Planning  Guide  (Wash- 
ington, 0.(2.:  U.S.  Department  of  Labor, 
April,  1974). 


U.S.  Department  of  Labor , ^Employment  and  Training  Administra- 
tion ^Comprehensive  Employment  and  Training  Act  Regulations; 
Final  Rule   (Washington ,  D.C.*  Federal  Register  Vol.  45,  No.  99, 
May  20,  1980).  nV-51    "  '«*    .        "  * 
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Step  3 


Review  Programs  and  'Plans  of  Other  Agencies/Service 
Deliverers/Businesses 

Thfe  purposes  of  this  step  are  to  pinpoint:    gaps  in  the  * 
local  service  delivery  system;  possible  arjeas  for 
collaboration  with  other  service  deliverers;  and  areas 
in  which  your  PIC  may  choose  to  initiate  new  services 
or  projects . 

The  step  consists  oif  identifying  all  Qf  those  programs 
which  potentially  impact  upon  employment  and  training 
services  and  your  PIC's  program  and  then  looking  at,  tne^r 
past  performance  and  future  plans  For  service^  Programs 
that  you  should  Review  include:     the  CETA  program,  Ebploy-  * 
ment .Service  programs,  economic  development  programs,  govern7 
mental  agency  .programs ,  educational  programs;  relevant  private 
^ecto^ ^^^^^h^ 


Pr-ime  Sponsor  (PS)  Annual  Plan:     document  which  Prime 
Sponsor  prepares  annually  to  describe  program  for  pro- 
viding activities  and  services  to  the' eligible  popula- 
tionr-for  upcoming  fiscal  year  and  to  apply  for  federal 
assistance. 

V  . 

Subparts:    major  segments  of  the  PS1  Annual  Plan. 


i 


m 


fl(  \[    Knofc'ieaoc/CapabiHtiss  bating 


Review  the  lpffcrmation  provided  above.     Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  this  step  on'the  scales  below. 
•  « 

•  Knowledge 

•  C^pabiliti^s 


1  

i 

1 

I 

Excel- 

- Above 

Aver- 

Below 

Poor 

lent 

Avg . 

age 

Avg, 

'     I  * 

1 

1 

I 

J 

Excel- 

Above 

Aver-  > 

Below 

Poor 

lent 

Avg. 

age  , 

Avg. 

• 

Based  upon  your  ratings,  decide  whether  you  'need  to,  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the 
next  step". 
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! PME  PROCESS  NOTES/ ACT I ON  PLANNING  PAGE 


NOTES   (Points  (that  are  important  which  ^oir  want  to  remember) 


7 


QUESTIONS   (Items  on -which  you-  ar,e  unclear  or  want  to  get  additional 
1        "  information) 


ACTIONS   (Specific  steps  that  you  intend  to  take  to  implement  your 
PIC's  PME  process  in  this  area) 


steps 


Resources/Persons 


Time  Frame 


\ 


r 
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v       INTRODUCTION  ^ 

The  CETA  regulations  state  that,  "The  PIC* should  consult 
with  the  Prime  Sponsor  and  its  planning  council  during  the 
development  of,  and  shall  have • the, opportunity  to  review  and' 
comment  on,  other  Annual  Plan  subparts  under  the  Act."  They 
also  recommend  that  the  PIC  should,  "to* the  extent  possible," 
ensurte  that  its  plan  is  consistent  with,  and  "where  possible," 
review  and  comment  upon  plan,  and  grant  applications 'related 
to  private  sector  employment  and  training  which  are  submitted 
to  other  federal  agencies  for  funding.  .  - 

The  fundamental  reason  N^hat  this^  reciprocal  planning  ne- 
view  .process  is  set  out  in  th$"  regulations  is  to  ensure 
coordination  of  employment  and  training  planning  at  the  local 
level.     However,  you  are  probably  less  concerned  with  what 
those  requirements  mean  in  terms  of  abstract  concepts  such 
as  "coordination" -  and  are  more^ concerned  with  what  they 
mean  specifically  in  terms  of  what  you  ought  to  do  as  part 
of  *your  PIC  planning  process.' 

s  A.     MAKING  OTHERS 1   PLANS ^WORK  FOR  YOU 

Actually,  the  review  of  other  agencies'  or  service 
deliverers1  plans  can  provide  ypu  with  data,  programmatic 
approaches,  and  ideas  which  may  be  extremely  useful  in 
helping  to  shape  yourPIC's  plan.     Specifically,  by  review- 
ing others'  plans  you  may  be  able  to: 

•       secure  timely  data  regarding  target' 
group  and  local  labor e market  need 


6 


•  ■    identify  programs  which  have  a  high 

-success  rate 

•  •   pinpoint  potential  service  deliverers 

who  provide  services  efficiently  and 
effectively 

•  determine  who  is  going  to  do  wKat  to 

•  K  •  'help  whom,  > * 
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Based  upon  this  feview,  you  can  structure  lyour  PICf s  pro- 
gram to  complement  ank  reinforce  other  efforts  /being  uiylertaken 
at  the  local  level.    fHow  you  do  that  will  vary  J  according  to 
condibipns  within  your  employment  and  trainingjcommunity .  How- 
ibilities**  would  ihclude: 


ever ,  possibi] 


directing  JPIC  programs  toward  target 
groups  and potential  job  opportunities 
.not  being  (addressed  through  pjther 
programs  I  I 

"piggy  backing"  by  using  BIC  funds  ?to 
support  arograms  which  have  at  recopd  of 
demonstrated  success 

cooperating  or  linking  up  with  other 
agencies!  to  "joint  venture"  p.  p'rbject 

'        ;  L 

selectinjg  successful  basic  program]  de- 
signs ai}d  modifying  them  as  appropriate 
for  implementation  through-  tjhe  PIC 

identifying  employment  service  are  a?  .in 
which  there  are* gaps  and  developing  special 
programs  to  fill  those  gapsj 


B.     REVIEWING  THE  PRIME  SPONSOR'S  P 


Obviously,  the  most  important  plan  in 


v^ich  to- provide 


input  and  to  revjiew  is  the  Prime  Sponsor 1 s| Annual  flan.  The 
Annual  Plan  consists  of  an  ag^Lcation  for-j  federal!  assistance, 
a  narrative  description  of  programs,  Progrjam  Planning  Summaries, 
Budget  Information  Summaries,  and  otjher  fqrms.     Tlje  Plan  itself 
is  structured  ap  follows: 

A. 
.B. 
C-L. 
C. 


D. 


Application,  for  Federalj Assistance 
General  Annual  Plan  Nafjrative; 
Annual  Plan  Subparts  j 

Ti)ble  II,  Parts  B  &  C:  ■  Trairiing^Act  Jvities , 
Upgrading  and  Retraining 

Tojtle  II,  Part  D:     Public  Service  Emfc>loy- 

m0nt  and  ClaSsroom  Training 

i  , 
title  IV,  Youth  Community  'Conservation 

Improvement  Projects  / 
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F.  Title  IV,  Youth  Employment  ahd  Training 
Programs  •  •  ■  -^  , 

G.  Title  IV,  SuirrniefT  Youth  Employment  Program  • 

H.  Title  VI,  Public  Service  Employment         •  * 

I.  Title  VII,  Private  -Sector  Initiative  Ptogram, 

J,1  Title  III-,  CETA  National  Demonstration  / 
Programs    ,  " 

K.     Administration  Annual  Plan  Subpart 

L.     Planning  Council  Recpmmendations  ;and 
Governor's  Comments^        /      f  . 

M.     Required  Annual  Plan  Forms  .  t 

1 .     Budget  Inf 6rmation • Summary 

BIS .  Backup  -        ^  *  « 

CETA  Program  Planning  Summaty  "  \ 

Summary  of  Subrecipients  an<?  Contractors 

CETA  Monthly  Schedule^  "  -•' 


2. 
3. 
4. 


Those  portions  of  the  Annual  Pldn  which  are  mo^t  worth 


reviewing  in  shaping  your  PIC's  Annual  P]|an  .are 


•  General  Annual  Plan  Narrative  -  summarizes 
^information  obtained  through  monitoring,  and 

-  evaluation  of  all  programs  under-  the  Com- 
prehensive Employment  and  'Training  Plan* 
(CE^P)  and  outlines  ^annual,  plan  of  service 

•  to  target  groups  from  significant  demo- 
graphic segments  in  the  Prime  Sponsor  area. 

#  Title  II,  Parts  B,  C/  and  D  -  describes  . 
varipus  training,  upgrading*  and  retraining 
programs  to  be  sponsored  through. the  Prime 
Sponsor. 

•„  Title^IV,  Youth  Employment  and  Training  Prbgram  - 
details  activities  and  services  to  be  greeted 


at  young  persons  within  the  community. 


narrative  description  in  the  Prime^Sponsor 1 s  Mastet  Pi 


.ew 

h 


In  addition,  you  might  fi,nd  it  worthwhile  to  tevi^w  the 

jtet 

The  Master  Plan  serves  as  the  long-term  agreement' on  basic 
program  planning  and  administrative  matters  between  \^the 
Department  of  Laborvahd  the  Prime  Sponsor*     It  has  n&  pre- 
-established expiration- date.    Once  developed,  it  is  nipdified 


ERIC 
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only  as  necessary *to  maintain  accuracy.*         *  ,  • 

The  Master' Plan 'is  a  potentially  important  source  docu- 
ment  for  your  consideration  as  a  PIC  because  it  contains  basic 
-data  and  Information  on: 

c 

"  the 'geographic  area,  population,  and  labor    .  I 
market 

overall  goals^bf  the  PS\  program 


*  J* 


basic  program  approaches  and  a  description 
of  arrangements  to^ensure  the  provision 
off* employment  and  training  services 

descriptions  of  procedures  and'  criteria 
for  aerate©  deliverer  selection 

*  >  u  #  <■*  * 

mechanisms  for  coordination  and  linkages 
yith  thef  St^te  lEniploymen^  Security  Agencies 
.  and^  other  Federal*  agencies  '  ^    *  - 

a  description  of  the.  Prime  Sponsor's 

planning  procedures  and  responsibilities  . 

v 

a  description  of  the  Prime  Sponsor1^ 
organizational  -Structure  , and  staffing,  : 


•    *    '-Jm   "    *'  C.     REVIEWING  OTHER  P^ANS       ,  •  ■ 

Other  agencies-  vftose  p^ans  should  be  reviewed  at' thij^l 

<  *      '    *  '     *  *  '* 

stage  include:  .  - 

•  '*     .  ,   -  Y 

State  Employment  Security  Agency  -  Program  Xs 

and  Budget  Plan 


Economic  -Development  Administration  _ 

Department  oX  Housing  and  Urban  Development' 

,§^i;ft*>*»iqess'  Administration  •      f     .  , 

Community  Services/ Administration 

In  Addition,  as  tinje  allows,  you  shpuld  attempt  to  review  the 
plans  and  program  performance  of  all  those  state  and  lofcal  - 
goverwjteht  .agen^es  which  provide  Services  of  .an  .employment  . 
and  *  training  nature  within  yyur  local-  commvffiity.     Finally,  don't 
ignore  'anywork  that  private^ector  firms  m^y  be  do^ng  ^frich* 


alfco_might.be  relevant.    Given  the  large" number  of  firms, 
*hir>ill  be  difficulty  but  as  you  begin  \q  focus  on  growth 
occupations*  it  becomes  very  important  to  identify  'the  kinds 
-of  training  and 'the'  types  of  facilities  available  in  the  m 
private  sector,    You. -may  bg  ablelo  build,  on  what  employers 
are  already  doing,  use  their  staff  'or  facilities  for  a 
nbrtinal  charge  or  for  .free, 'or  follow  theireurricuiaJ 


CONCLUSION 


In conclusion",  looking -at  the  plans  and  performance  of # . 
0tj30^nCieS  S*T2       deliverers  will  help  you  tcj 

det*rl(Tne  how  to  -construct  your  PIC  program  so  that  it  fits  ' 
>*t  easily. within  and  can  be  most  effective  for  your- community. 
To  ensure  this  responsiveness,  ^to  the  extent  possible,-  the  '  # 
-Council  should  be  involve^  in  reviewing  and  reacting  \ to  the 
comments  and  plans  of  other's.      "  ' 


Manpower  Administration  (now  Employment 
.and  Training  Administration  [ETA] J ,  ' 
Manpower  Program  Planning  Guide  -(Wash- 
mgton,  d.C:  U.S.  Department  -of  Labor, 
April,  1974)  .  .  ' 


U.S  .  Department  of  Labor  , .  Employment  and 
Training  Administration  Comprehensive  ' 
•Employment  and  Training  A&  Regulations 
Finar.  Rule  <Washiigt6n,  D.d.:  Federal 
Register  Vol.  45,  No'.  99,  May  20,,  1980) 


r 


B.     DEFINE  PIC  STRATEGY 


Purpose: 


To  develop!  using  the  data  secured  in  the 
npeds.  assessment  and  an  analysis  of  the  PIC's 
programmatic ,  organizational  and  administrative 
performance  over  the  preceding  year,,  a  strategic- 
plan  consisting  of  goals  to  be  accomplished  in 
the  upcoming  year  and  other  goals  to  be  accom- 
plished in  the  lontf  term. • 


Steps 


1*%-Asse$s  PIC  Program  Performance 

2,     Assess-^IC  Organisational- and 
Operational  Performances 


4k 


V 


3.  •  Review  Needs  Assessment  Findings 
a.     Define  EGS  Need^ 

Select  Target  Groups 


c. 


Determine  Occupational 
Training  Priorities 

Define  Participant  Service 
Needs 


Assess  PIC  Program  Performance 


The  purpose  of  'this  step  is  to  determine  how  weld  a  PIC 
performed  programmatically  ^i.e. ,  in  terms"^*^pro ject  or 
service  activities)  so  that  information  c^n  bemused  in 
replanning.     In  this  step,  you  look  at  prdgram^perform^nce 
at  three  levels — overall ,  by  component ,  and  by  service 
deliverer,*     Data  obtained  from  the  PIC's  evaluation  system 
provides  the  primary  inputs  for  accomplishing  the  step.  . 


v 


# 


y  •  Planned -versus^  Actual  Performance;  measurement  of 

i  (   ment  o^  -accomplishments  against  pre-set  objectives  and  - 

'   standards  contained  in  annual  plan;  M «  > 

•  Deficiencies  or  Significant  Variations:     areas  in  which  * 
.  performance  falls  significantly  below  tjhat  planned. 

*.  *  -  * 

•  •  Blockages/Barriers :     obstacles  or  reasons  for- less  than 

)    successful  performance  in  an  area, 

v  •  Components r    major  specific  activities,   including  training 
and' other  participant  programs  and  employment  generating 
services.  ^  K  fc  *' 

•  Service  Deliverers:     any " organization *or  finh  the  PI^  uses- 
to  provide  services  or^perform  activities*  1 


i 


§R  j(   mowfedoe/Capabiinies  Rating       >  Lj^^ 


Review  the  information  provided  ab*ve,.  Rate  your  ,  P,IC 1  si  knowledge  'and 
capabilities  with  reference*  to  this*  step  on  the  scalesi/eelow.  t 


V 


•  Knowledge 


f  Capabilities 


Excel- 
lent 

m 

L 


/£>ove 


Aver- 
age 


■fee low/ 
Avg . 


ft 


eric; 
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v. 


1&2 


Poor 


Poor 


^£xcel-  fcbove"         Aver-  Belc 

.   'lent-     *       Avg.  age  Avd,  , 

Based  up6n  youf  ratings/  decide  whether  you  need,  to/rgad>*fee  content 
inficrmation  on  this  step  whichxfollows  immediately  br  procrdBd  to  the 
next  step.  *    ,        i .  C  * 
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PME*  PROCESS  NOTES/ AQTION  PENNING  PAGE  ^ 


NOTES  (Points  that  are  important  which  ypu  want  to  Remember) 


QUESTIONS   (Items  on  which  you  a^e  unclear  or  want  to  get  additional 
information) 


ACTIONS   (Specific  steps  .that  you  intend  to  take  to  implement  your 
PIC'svPME  process  in  this  area) 

\  \        f[      V Resources/Persons  Time  -Frame!  , 


Steps. 


ERJC  - 


l$3 


ERLC 


INTRODUCTION 


As  nqted,\your  PIC's  evaluation  system  should  furnish 
the  information  that  ^oa  will  n^eA^  for  performance  refView; 
If  you  have  an  adequate  system  in  place,  that  system  will 
provide  you  withlalmost  all  o£  the  objective  data  thaft  you 
need  to-  review  y<Vr  PIC's  performance  in  ifeuch'a^way  that 
it's  Relatively  ea\y  to  make  decisiqns  based  upon  that  L 
review.     Section  F,  "Develop  Evaluation  Plan;"  describes* 
how  to  put  a  comprehensive  evaluation  system  into  place  for 


future  use f 


Let's  suppose,  however,  that  your  PIC  doesn't  have 
the  perfect  evaluation  system  right  now.    What  can  you  do 
as  an  interim  measure  to  review  your  PIC's  performance 
in  9  systematic  manner?  '  . 

First,  realize  that  you  don't  have  time  arid  it's  not 
necessary  to  look  at  everything.     Therefore,  you  have  to 
decide  what's  important  to  look  at,  wjiat  questions  to  ask 
what  methods  to  employ,  and  what  measures  or  indicator? 
to  Use  as  means  to  assess  performance. 

• ' 

A.     DECl-DINg  WflAT  TO  LOOK  AT 

There  are  a  variety  of  areas  that  can  be  looked  at  in 
reviewing  yout  PIC"s*-i5^ogram  performance.     These  include i 

•  '  accomplishment  arid  appropriateness  of 

organizational  purpose  and  goals 

•  delivery  of  program  services  .and.  other 
activities  (EGS) 

•  '  relative  effectiveness  of  various 

service  deliverers 

jfs  -    #       the  overall  impact  of  ,the  PIC's  * 
prpjects.  m 
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B. 


TERMINING  THE  QUESTIONS  TO  ASK 


^  The  areas/ypu  fchoi>se  to  look  at  and  the^ factors  you 
define  as  yaportant  Will  determine  the  questions  that  you 
need  .to  ask  regarding  your  PIC's  performance.     Some  possible 
questions  which  ^rou  might  ask  as 'they  Relate  to  various . areas 
.oft^naiysis  follow.  - 

Economic  Impacts  c n  Participants 

#-What  <^o  various  measures  of  placements  and 

terminations  show? 
•  What  do  various-  Measure?  of  wage  gains  show? 
.  •  Wh§^t  can  rba -said  About  the  quality  of  jobs 

developed?     •  .  V 

f    '~m  Wha't(jai>  be  said  about  the  retention  of 
*     participants  in  the  jobs  they  obtain? 

>  Ttfbat^can  be  s^id  ab<rat  advancementriDT'par1 
'ticipants  from". their  first  job  to  better  jobs? 

stitutipnaj.  -Impact  *of  Title  VII 


PIC  stimulated  new  business  partici- 
in  local  employment  and  trcjining  efforts? 
'  PlOr-^rtimfilated  new  responsiveness  on 
rt  of.  "traditional"  employment  and 
t;rai*iing*  deliverers  '  to*  the  needs  of  business?  /' 
-   •  Is.  ttrere  a  comprehensive  pljannlrlg  approach 
•    -  *      "fcetveen  PIC  an<J  th^ 'local  Employment  and 
•^^^  \      Training  Council'?  / 

,  •  9c*s  Xhe  P'iC  stimulated  any  areawide  planning 

*  \'    'between  d\ f  feren't  'psime  'sponsorship  areas 
.  ~  'and 'other  PICS?  ;. 

I  k  .  "  '  • ' 

Afctitudijial'  Impact  of  ,Tit»lg  V13 

0   How  has  the  PTC  jprbga/am  -,af  f ected  the  attitudes 
4  -  ^.fit  ejuptoyers  toward  ^the  program,  -toward  employ- 

-TTienf  and  training  programs  in  general ,  and 
'  \  \   ^Jtovlard  eliqible  clients  as  potential  employees? 
#fWfcat  impact  haa  t£i$ -Pie?  program  had  on  the  atti- 
'   t  tudles  and  mot iv&tJions  <jf  participants? 

f      '         —  - "..  '     "*   \  ' 
Patterns' of  Service     *   \     }  % 

#  Vfkat  ape  tli*  •character isfciefe  of  the  partici- 
belng  served?    .Hoy  do  they  compare  to 
X  ^CEtjk*gll6ntsr  goal's,'  and  some  state- 
wt'of  "universe  of  *n%&cJn}#  *       .  "  ; 

' Wh&t-  are  the  characteristics  of  the* businesses 
£6irrg  setVed?-  WJieire  'do^they  stand-in  the  • 
•overalls  Economic  t  structure  of.  the  area? 


v  *^wh4t  reasonable  c^st-zbejief it  calculations 
Ip  „        •  ca|i  b^jff^d'e  a 


Sut  some 'of  -the  above  ^lmpactk?- 


4- 


C.     EVALUATING  PROGRAM  OUTCOMES 

As  -the  foregoing  listing  suggests,  the  options  avail- 
able for  a  summary  review  of  performance  are  numerous.  Atfc 
a  minimum,  your*  PIC  will  probably  want  to  answer  four  basic 

questions  regarding  program  outcome: 

%  Hoy  well  did  your  PIC  perfonr.  numerically 
\  in  terms  of  participaftts,*  costs,  resource 

S utilization?  *  ■ 

•  Was  your  total  program  implemented  , 

according  to  plan1-  qualitatively? 

^  •  }  •  g 

•  How  well  did  each  component  (e.g., 

_  _  classroom  ^ttain^ng,  marketing,  labor   ( 

"""market  analysis)  perform?*' 

•  Wh£t  was  the  relative  effectiveness 
or  performance  of  the  service  djeliv- 

,  erers*  you  .used?  J 

t      '  * 

Answering  the  questions  will  enable  you  to  identify  % 
areas  in  jw^ich  there  were  performance  shortfalls,  and  to 
follow  up  to  ascertain  the  reasons  j(blfc>ckage^/barriers)  for 
that  shortfall.     This  level  of  inquiry  will  also  enable  yott 
to  draw  conclusions  which  can  be  used  to  make  basic  re- 
planning  determinations  suoh  as:         +    "  . 

'   •  Were  your  initial  planning  assumptions  4 
r"valid?  /;  * 

f  i'Were  your  initial  goals  and  objectives  « 
^      .realistic?  ~ 

•  Was  your  overall  programmatic  thrust,  or 
"    mix,  of- services  and  projects  sound?.,    \  ^ 

■3 

•        r  •  Are  there  any  components  of  your  program. 

which  should  be  dropped?  Given  increased 

•support?    '  '                  \  \                 ,  * 

» 

m  'Are'  there  any  service  deliverers  who  are  *  . 

ineffective?    Highly  effective? 
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D.     USING  THE  MOST  COMMON  METHOD: 


VERSUS  ACTUAL  PERFORMANCE 


There  is  on^  basic  method  that  Ls  used  more  often  than 
any  other  for  outcome  evaluation.-  That  is  planned  versus 
actual  performance. 

There  are  probably  three  reasons  for  the  dominance 
of  this  method  in  performance  review: 


l 


•  iThis  type  of  review  tells  you  how  well  ^ 

your  program  did  "compared  to  objectives 
or  standards  that  you  established  for 
yourseif  at  the  beginning  of  the  yeai;. 

•  '      The  majority  of  data  for  this  form  of 
*   revilw  can  be  retrieved  easily  from  * 

reports — Program  Status  Summary,  Summary 
of  Participant  Characteristics,  and 
•   Financial  Status  Summary — which*  are  pre- 
v^  pared  quarterly  and  annually  for*  the  „ 

Department  of  Labor.  - 

•  /      The  performance  versus  plan  evaluation 

approach  is  that  traditionally  used  by 
the  Department  of  Labor  in-*  its  assess-  • 
ment  of  employment  and  training  program  ♦ 
^  performance        *  *     ^ » 

*  • 

Variance  .  - 

The  concept  whifch  is  central  to  the  quantitative  measure- 
ment -of  performance  against  plai)  is  "variance."  Variance  is 
the  proportion, -  expressed  in  a  percentage,  of  the  dif ferencg^ 
between  planned  performance  and  actual  performance.  This 


fact  is  expressed  in  the  formula  below: 

\  ~-  -   v-   ^  € 

For  example,  if  the  planned  total  enrollment  for  the 
yea^r  *i%  your  PIC's  classroom  trainmc||program  was  50  and  the 
actyal  enrollment  was  30,  the  variance  would  be  commuted  as- 


•  •■'  '  •/*-  is?  ■■ 
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showrf  below 


30-50 
50 

-20 
50  * 


100    -  variance 


100    =    -40%  variance 


Calculating  variance  allow?  you  to  state  exactly  how 
well  your  program  did  as'opposed  to  plan.  v Although  there 
is  rp  hard  and  fast  ruleT  a  variance  of  +  15%  is  normally  i 
considered  tolerable.    A  variance  of  25%. normally  indicates 
that  there  may  be  a  problem  in  the  area  in  which  th^re  is  a 
variance. •  * 


Goal/Objective |  Attainment  4 

*A  second  ccJncept  wffich  is^key  to  performance  versus^  ^ 
plan -evaluation  is  "goal  or  objective  attainment."  This* 
concept  is  most  ujseful  in  the.  measurement  of  less  quantita- 
tive program  outcomes.     The  concept  is  used  to  measure  the 
degree  to  which  PIC  objectives  IS^r^  realized. 

Factors-  vhich  are  nontfally  taken  into  account  in  16oking* 
at  goal  attainmeilt  include: 

•  timeliness  . 

•  quantity  , 
r  •  quality 

».      cost  efficiency.  4  / 

/•The  concept  of  goal  attainment  is  explained  furthe^ 


under  the  next  heading  *on 
Plan  Perspective. 


'Applying  Performance  Versus 


+ 


ER?C 


E.     APPLYING  THE  PERFORMANCE 
VERSUS  PLAN  PERSPECTIVE 

f 

reiterate,  there  are  fous  essentieLL  areas  for 

'  '  ^  --\  #^ 

iAing  your  PIC's  program  perfotAance: 

•  *   '         »  •* 

Overall  Quantitative  Analysis  * 
Overall/Component  Goal  Attainment  Analysis 
Component  Review  and  Analysis  # 
Service  Deliverer  Review  and*  Analysis. 


! 
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OveAall  Prograj^Performance  Review      Quantitative  Analysis 
1  \  For  purposes  of  a  numerical  assessment,  the  aggregate 
p^rfcrmarfce  of  the  PIC  can  be  broken  into  two  major  cate- 
gories:    direct  services  to  participants;  and  other  services 
and  activities.  4  » 


1 .     Direct  Services  ta  Participants 

^Tiiere  are  a  variety  of  outcome  indicators  that  you  can 

use  tollookat  direct  services  to  participants.     The  five 

principal  areas  in  wfoich  measurements  of  performance  are 

normally  made,  however-,  are:*, 

service  levels  by  target-' group 
.enrollment  level  «.  y 

expenditure  level  , 
^results  achieved  eft  end  of  service 

(termination)  v  \ 

outcomes,  in  relation  to  toost.  \ 

Simdle  format^  that  £qu  can  imploy  to  review  and  analy^ 
performance  against  plan,  as  determined  through  variance  for 
each  of  the  areas  cdied  above,  are  presented  on -the  following 
pages.'       *  •  * 


PERFORMANCE  VS.'  PLAN:    SERVICE  TO  TARGET, GROUPS 

# 

I 

i 

%  of  Total  -Participants  - 

%  variance' 

GftOHP 

ACTUAL   *  n 

PLAN 

SEX 

Male     .  ^  . 
,  Female  \ 

 »  1  

RACE 

■  \ 

WhVte/Non- 
Hispanic 

Black/Non-  f 
Hispanic  ' 

Amer.  Indian  or 
Alaskan  Native 

m 

Asian  or, Pacific 
Islander 

* 

AGE 

■ 

19  and  under 
20-21- 
22-44 p 
45-54  ' 
55-older 

I 

SPECIAL 
CATEGORI-tS  • 
(as  defined 
by  PIC)  •  . 

t 

•  Handitapped 

Vietnam  Er$ 
Veterans  ^ 

ADC  Recipients 

.  Ex-Offenders 

Others  (list) 

*  > 

ft 

4 

• 

i 

t 

/ 

-  « 
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PERFORMANCE  VS.  PLAN;  ENROLLMENT 

 1  — c  _L 

AND  EXPENDITURES  '. 

INDICATOR 

— - — t 

RESULTS 

ACTUAL 

PLAN 

%  VARIANCE 

.  Total  Cumulative  Enrollment 
.  TotaT  fcrrent  Enrollment 
.  Jotal  Accrued  Expenditures 

i 

  C 

t 

PERFORMANCE  VS.-  PLAN:  TERMINATIONS 


INDICATOR 


FORMULA 


RESULTS 


ACTUAL 


PLAN 


%  VARIANCE 


Positive  Termination 
Rate 

^Entered  Employment  • 
Rate 


Indirect  Placement 
Rate 


.  Private  Sector 
Placement 


T3t:  ^Pos .  Term, 
Total  Terror 

Tot.  Ent.  Emp. 
Tot,  Term. 
Less  Transfers 

Tot. Indir. Place. 
Tot.  Term.  - 
•  Less^  Transfers 

Priv'.  Sec.  Place. 
Tot.  -  Entered 
Emplbyment 


x  100 
x  100 

x  100 

x  100 


I 


-  »   Z  =  1  ;  ■  

^ PERFORMANCE  VS.  PLAN:    COSTS  # 

INDICATOR 

FORMULA 

— —  1 

ACTUAL 

RESULT5 
PLAN  ' 

%  VARIANCE 

.  Cost  P^r  Positive* 
Termination 

.  Cost  Per  Entered 
Employment 

.  Cost  Per  Indirect 
PI acement  «? 

Tot.  Accrued  Exp.'  ! 

/    -,-.-1  - . 

f  ■  

-  j 

m 

t 

Tot.  Pos.  Term. 
Tot.  Accrued  Exp. 

*. 

Tot.  Entered  Emp. 
Tot..  Accrued  Exp.  »"  • 

Tot.  Indir.. -Place ;>  fc 

*                        T   - 

* 

In  conclusion,  measuring  in  the  foregoing  areas,  using 
the  formats  provided  or  substitutes  more"  appropriate  for  your 
situation, ,will  give  you  a  good  picture  of  the  overall  quan-  ^ 
ti€«tive*results  achieved  through  the  direct  service's  to  par- 
ticipants portion  of  your 'P^' s  programs- 


2.     Other  Services  and  Activities 

0       It's  much  more  difficult  to  do  a  quantitative  assessment 
fcg?  those  services  and  activities  which  are  not  directly*  ^ 
participant  related.     It  is  possible,  however,  to  do  a  simple 
analysis  of  varianbe  of  costs  for  those  nonpartici]kant  ser- 
vices undertaken  by  the  PIC.     The  chart  below,  or  a  similar 
one,  could,  be  used  for  that  purpose.  ,  w 


PERFORMANCE  tfS.  PLAN:    OTHER  SERVICES  AND  ACTIVITIES 


RESULTS 


SERVICES  AND  -ACTIVITIES- 


ACCRUED 
EXPEND. 


ACTUAL 


$$  BUDGETED 


PLAN 


%  VARIANCE' 


Administrative  Cost Pool, 
Intake,  and  Assessment 
Employment  Generating  Servs.' 
Other     .  •    -  ; 


.The  more  important t aspect  in  an  analysis  of  these  services 
would  be  to  review  them/within  the  context  of  the  implementa- 
tion of  your  PIC's  total  plan.  * 
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program  Performance  Review  -  Goal  Attainment  Measurement 
Of  necessity,,  the  process  of  evaluating  your  PIC& 
success  in  implementing  its  total  plan  must  fcfe  somewhat  mbre 
subjective  and  less  quantitative  than  that  which  can  be  J 
accomplished  In J looking  at  outcomes  which  involve  participants 
direotly.     HowAer,  that  doesn't,  mean  that  it  is  any  less 
important.    .In  fact,  quite  the  contrary  iar  true/  m  v 

Given  the  fact  that  Title  VII  is  a  demonstration  Titl'e 
which  encourages1  innovative  and  experimental  programs  and 
administrative  arrangements,  and  allows  your  PIC  to  devote  up 
to  3Q%  of  its  budget  to  Employment  Generating  Services—whicl^ 
can  be  non-direct  in  nature — the  need  for.,  additional  and 
different  forms  of  evaluation  beyond  the  strictly  quantitative 

is  critical .  -  %  - 

However,  you  can  devise  ,a  Simplified  performance  vs.  plan 
approach  to  review  performance  in  these  areas.     The  steps  . 
involved  to  do  this,   in  a  retrospective  fashion,  would  be 

as  follows: 

1.       Review  the  annual  plan.  t 

3.  List-all  major  objectives  or  components 
as  yet  unmeasured  (e.g.,  Employment       .  » 
Generating  Services;  to  incorporate 

%         PIC) . 

.    3*      Identify' primary "activities  to  be 
undertaken  with  reference  to.  each, 
area   (e.g.,  Employment  Generating 
Services  -  marketing  program;  Job 
•  '    Development  program;  Targeted  Job 
-Tax  'Credit  Program)  . 

4.  Specify  end  results/  produces,  or  * 
outcomes  expected  for  e^ch  activity 

•listed  (e.g.,  Jdb  Development  Pro- 
gram: job  fair  -  2  days  in  length, 
to  have  .representatives  from  100  *  % 

•    >  •  .  .    local  private  septor  firms,  to  pro-  * 

duce  .250  new, jobs  for  CETA-eligible  , 
participants;  Marketing  Program  - 
ffrint  and  distribute  3,01)0  brochures 
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to  4all  local  /industries  employing         ,  j 
between  500-525  individuals,  emphasising  . 
benefits  of  private  sector  involvement 
a*id  explaining  how  to  participate  in 
PIC  programs).  x 

•»  , 

5.    Once  the  foregoing  steps  arfe  accomplished, 
%    review  each  intended  result  in  turn  to  y 
identify", whether  or  not  the  outcome  was*  * 
achieved • 

i  • 
6*    For  every  outcome  not  achieved,  indicate 
.what  was  aqcompli^hed,- explain  present 
status,  and  identify  what  prevented' 
achievement • 

7.    xFor  every  joutcolie  which*  waar  achieved  ,t 
assess  th^l  f i^al  prodact  to  ascertain 
whether  a t\ conformed  to  specifications 
or  'expeg^yitons  regarding: 

-  J  c/r  .quantity 

^^^^  ^  t 

*     .  8.  %Basdd  upon  your  ra'fing»of  each  completed 

outcome' according  to  the  criteria,  assign 

cf  performance  level  to  tbe  outcome: 
•  <* 

-  Greatly  Exceeded  Expectations>6Becif  ications 

-  Exceeded  Expectations/Specifications 
-    <                 -  Met  Expectations/Specif ic^tionsV 

.    a  "  Did" Not  Meet  Expectations/Specif  motions 

Fell  Far  Short  of  Expectations/Specifications 

9.     As  for  outcomes  not  achieved,  for  those 
outcomes  which  did^not  meet  expectations 
or  specifications,  identify  if  possible, 
.what  factors  might  have  caused  less  than 

expected  results. 

•  r~  ' 

Developing  a  chart  similar'to  that  presented  on- the  following 
page  will  facilitate  your  PIC1  s  implementation  of  the  performance 
vsv  plan  analysis  described  in  ^the  foregoing  steps.  .  * 


Cv. 
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Component ^Performance  Review  *         •  x 

Once  you  have  looked  quantitatively  and  qualitatively  at 
'tyour  total  program,  you  will  probably  next  want  to  look  at  "its 
major  component  pieces,  such  as: 

ClassroDm  Training  -  Occupational  Skill 
Classroom  Training1  -  Other 
-  OJT 

Upgrading       -  4  - 

Retraining 

Employment  Generating  Services  ..  1 

V  / 

Various  weighting  schemes  have  been  developed  wn±th  permit 
a  direct  comparison  'of  components  to  assess  their  relative 
""effectiveness.     These  approaches  aret discussed  in  trie  evaluation ^ 
section  of  this  Guide  and  they  may  prove  helpful  to*  veflu  in  the 


long  term. 

^  At  this  juncture,  however,  a  more  pragmatic  approach  is  to 
look  at  your  'program1  s  components  by  applying  the  sarte  quantita- 
tive performance  vs.  plan  perspective  that  you  usgfci  to  look  at 
your  program  in  the  aggregate.     Useful  indicators  again* would  be: 

•       cumulative  enrollment?  compared  to  planned 
capacity      .  ( 
■  accrued  expenditures 

positive  termination'  rates 
*  entered  employment  rate 
cost  per- positive  termination 
cost  per  entered  employment  , 

Chart's  which  could  be  employed  or  modified  to  fa'^ilitate  com- 
parative analysis  of  components  follow. 
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PERFORMANCE 

VS.  PLAN: 

ENROLLMENT  AND*  EXPENpiTURES  • 

 ~i 

 /  :  >  

* 

1  1 

COMPONENTS    .  J 

CUMULATIVE  ENROLLMENT 

CURRENT  ENROLLMENT 

ALLKU 

:U  tArtnUl 

ti  tore 

Actual 

Planned 

%  Var. 

Actual 

Planned  ' 

f%  Var. 

Actual 

Planned 

%  Var. 

Classroom  Trng.  -  Occ. 

.  i 

Classroom^Frng.  -  Other 

wr\ 

Upgrading 

Retraining  , 

EGS 

;  F 

Other              "  t 

Other 

I 

-J 


PERFORMANCE  VS.  PLAN:*  TERMINATIONS 


COMPONENTS 


Classroom  Trng'.  »  Occ.  1 


CUssrooni  Trng.  -  Other 


OJT 


Upgrading 


Retraining 


EGS- 


Other 


Other 


POSITIVE  TERM  F 

IATE*  ' 

ENT 

ERED  EMP.  R 

ATE* 

INDIREC1 

•  PLACEMEN' 

r  RATE* 

Actual 

Planned' 

%  Var. 

Actual 

Planned' 

%  Var. 

Actual 

• 

Planned 

*  Var 

— > 

> 

* 

♦ 

> 

i  • 

• 

 1  

• 

* 
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*Select  indicators,  based  upon  nature*of  your  prograrrwhich  are  applicable  to  components 


COMPONENTS 


CVassroom  Trng.  -  Occ./ 


CI as&r,pom  Trng.  -  Other 


OJT 


Upgrading 


Retraining 


EGS 


00 

o 


Other 


Other 


PERFORMANCE  VS.  PLAN:  -COSTS 


COST  PEB  POSITIVE  TERM* 

COST  PER  ENTERED  EMPA 

r       COST  P 

Ik  PARTICIPANT*  m 

Actual 

Planned 

%  Var. 

Actual  • 

y  Planned ' 

%  Var.  v> 

J  Actual . 

planned 
— 3 — -  1 

%  Var. 

/ 

*  • 

/ 

-T- 

*Select*  indicators;  based  upon  the  nature  of  your  program,  which  are  applicable  to  comments.  .  %, 


20\ 


V 


202 


Service  Deliverer  Review  *v 

r  The  last  area  to  look  jat  in  a  summary  review  of  per- 
formance is  the  relative  performance  and* effectiveness  of  the  . 
'individual  service  deliverers  with:  wftich  you  contracted*. . 
Again,  rating  systems  have  been  developed  to  permit  a  straight-  ^ 
forward-  comparison  across  components  and  service  delive>rpr<=t .  r 
Maintaining  the  perspective  we  have  H*ad  tip  to  this  pointy  how-  . 
ever, .the  mqst  immediately  useful  forpi  of  assessm^rtt  of  service 
deliverers  is  against  their  plans,  and  then  to  comparS'  those 
deliverers  whicVare  providing  services  within,  the  sams  com- 
ponent   (e.g.,  OJT,  or  Classroom  Training.)." 

..This  comparison  is  most  useful  when  you  havevtwo  providers* 
furnishing  the  same  service  (e.g.,  clerical  training).     But,* it 
is  still  of  sOmeovalue  to  identify  the  relative,  costs  and  oift- 
.comes  associated  with  various  programs  in  order  to- make  trade- * 
bf^fs  and'  reach  defcisions  forvfuture  planning.     Using  a  ^c&art 
'similar  to  the  .one  on  the  followiag  page  would  permit  easy  review 
and  comparison  among  service  deliverers  within  a  component. 


/ 
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COMPONENT: 


Classroom  Training 


•  4 

• 

*  • 

-  -              *    '  -* 

:'••<■  / 

V  *         •'     *  ' 

ft'*  3 

SERVICES  DELIVERERS 

-4 

ENROLLED'  *■ 

TRNG.  EXPENDITURES  % 

i  >P0S.  TERMINATIONS 

ENTERED  EMP. 

r 

Plan 

Act.  ■ 

Var. 

Plan 

Act. 

Var». 

Plan 

Act.  . 

Var. 

Plan 

Act-. 

Var. 

Project  A 

Project  B 

# 

/ 

-J  

project  C               •  *  ■ 

• 

-  Deliverer  1  * 

» 

-  Deliverer  2 

^  

— v- 

■| 

• 

* 

i 

|  Lxc- 

M 
00 


Another  inctic^tor  frequently  used  in  this  area  is  pre-CETA  earnings  vs^:post-CETA  earnings, 
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"  F  /   using  performance:  review  data  for  replanning 

As  indicated  earlier  in  this  section,  performance  review 

isfdone  to  ascertain  whether  y6ur  PIC  performed  as  well  as, 

#  *  m 

better,  or  worse  than  planned  so  that  that  information  can 

fee  used  in  reaching  informed  decisions  for  ongoing  program 
planning. 

By  taking  a  summary  look  at  your  PlC-r-at  the  program, 
component  and  seryice  deliverer'  level£ — y©u  can  pinpoint  - 
areas  of  significant  variation  and  possible  potential 
problem  areas.  4  ' 

Based  upon  the  identification  of  a  potential  problem, 
yoft  can  conduct^a  critical  examination  and  collect  data  in 
oVder  to  determine,  what  the  reasons  for  'deviation  from  the  plan 
"were,     fhis  information  can  be  used  to  define  the  problem 
as  clearly  as  possible  and  then  to  reach  preliminary  judg- 
ments regarding  changes  that  need  to  be  made  in  the  PIC's 
planning  assumptions,  program  scope,  basic  component 
Approaches,  o£  in  service  deliverers.     Decisions  made  at 
this  point  become  primary  inputs  in  shaping  the,  PIC's  pro- 
gram tor  the  next  fiscal  year.  r 


CONCLUSION  ' 

We've  concentrated  most  of  our  Attention  in  tfcis  step' 
oji  the  concept  of  performance 'versus  plan.    As  rioted  earliet, 
however,  that's.not  the  only  way  to  look  at  program  per- 
formance and  results..    Your  responsibility  is  to  .consider  . 
the  potential  alternatives  for  PIC  prograij^evaluati#n,  -to, 
match  them  up  against  your  needs,  an#  then  to  desigikantf  use 
the  review  process  which  is  most  appropriate  given  your  ■ 
present  situation.     (For  a  discussion  on  the,  full  range  of. 
evaluation  Alternatives  available  to  your  PIC,  refer  to 
Stage  F,  Step  1,  pp.    III-255  to  III-278.) 
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Some  cautions  ^re  in  order: 

•  Don't  Ipok  4t  tob  much:    Restrict  your -inquiry  to  those**- 
'areas  and  jQue^tions  whi/ch  are  most  essential  to  assessing 

overall,  performance.  ( 

•  Keep  your  jTssessirtent  framework  simple;  Select  indicators 
for  which.  yoU  can  easily 'obtain  data  and  methods  for  data 
analyses  y^ich  are  not  too  complex  in  design. 

'     *         »  ~-  m 

\ 

•  Plan  ^on  how^you  will  use  the  results  of  the  review: 
Determine,   in  advancf,  how  you  will  follow  up  in  order  .to 

.  pinpoint  reasons  for  sources  of  .problems-  Make  provisions 
for  iijcorpofca^ing^  yopr  findings *  into  replanning  your  PIC's 
program.    • *  •  .  < 


^aijpovier  Administration  (now  ETA)  ,  Program*. 
'  Assessment  Guide  (Washington,  D.C.  :  U.S. 
Department  of  Labor,  April,*  1974$  . 

*  f 
Mayor's  Office  of  Manpower,  City  of  Chicago, 
-  Planning  and  Evaluation  under  CETA- (Wash- 
mg^on,  D.C:  U.S.  Department  of  Labor,  (. 
January  1976) . • 
"V  - 

Olympus  Rese^arcli,  Self -Evaluation  of  CETA .  Manpower  Programs: 
ft  Guide  for^Pjritae  Sponsors  (Springifield ,  VA:  U.S.  Department^ 
0af  Conpnerce,  \$75)  .  - 

Van  Horn,  Carl,  Evaluating  PSIP  Programs  and  Activities:  A 
Guide  for  PIC  Members  and  .Staff  (Eagleton  Institute:  " 
Rutgers,  .the  State  University  of  New  Jersey,  1980). 

Ohio  State  'University,,  A  Formative  Evaluation  of  the  Private  . 
Sector  Initiative  Program  (Columbus,  OH:  Ohio  State  University , 

ldSl). —  — 


207 


111-84 


Assess  PlC  Organizational  and 
'     Operational  Performance""* 

The  purpose  of  this 'step _±s  to  determine  how  well  your 
PIC  perfQnned  organizationally  and  operationally.  This 
.step  is  taken  to  examine  basic  arrangements  and  methods 
of  functioning  to,  determine  their  utility  for  future 
program  operation'.'  >, 


•  PIC  Organization  -  structural  arrangements  and -division  of 

•  roles  and  responsibilities  among  PIC  members  and  staff. 

•  PIC  Operations  -  manner  in  which  a  PIC  functions  both 
internally  and  in  relatioo  to  significant  others  (e.g., 
-Prime  Sponsor  and  cbmmunity)  . 

PIC  Prodbdures  and  Methods  -  basic  administrative  processes 
and  guidelines  -for  managing  and  directing  PIC's  efforts. 

•  FIC  Priority  Development  Need  Areas  -key  organizational 
and  operational  areas  in  which  a  PIC  needs  to  improve  its 
performance.  ^ 


C  fl  Knowledge/Capabilities  fating  j 


j 

Review  the 'information  provided  above.  Rate  your  PIC* s  knowledge  and 
capabilities  wffch  reference  to  this  step  on  the  scales  below. 

•  Knowledge  ^  I  •  *  * 

.    Excel-  Above     • ,    Aver-         Below  Poor 

lent--         Avg.  •  age  Avg. 

•  Capabilities  I   i   \  J  1 

*  Excel-  Above  Aver-      .  Below  Poor 

lent  Avg.  age  Avg. 

<»  • 

Bas4d  upon  your  rakings,  decide  whether  you  need  to  .read  the  content 
information  on  this  $tep  which  follows  immediately  or  proceed  to  the  ' 
next  step.  • 
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PME  PROCESS  NOTES/ ACTION  PLAIJNINCj  PAG^ 


NOTES  (Points  that  are  important  which  you  want  to  remember) 


QUEST?ON^^7tem 

information) 


ACTIONS  (Specific  steps  that  you  intend  to  take  to  implement  your 
PIC's  PME  process  in  this  area) 


Steps 


Resources/Persons 


Time  Frame 
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INTRODUCTION 

■   *        ;  • 

I 

Assessing  your  'PIC1^  organizational  and  opesational 
performance  is  a  parallel  step  to  assessing  program  per- 
formance.    This  step  involves  two  parts:  * 


Examining  Performance  ■ 

Defining  Priority  Developmental 
Need  Areas  x 


'I" 


A.     EXAMINING  PERFORMANCE 

Area^t4^rt  you  should  look  at  in  this  step  include: 

PIC  Organization,  PIC  Operations,  PIC  Procedures  and  Methods. 

Possible  questions  that  £ou  might  ask  yourself  in  each  of 

these  areas  follow.  * 

t 

PIC  Organization 

i~     Are  PIC  goals  clear,  understood,  and* shared, 
by  members?  *- 

•  Are  the  PIC  members  clear /on  the  role  they  ^ 
wan4rv£he  Council  to  play  with  reference  to 
the  Private  Sector  Initiative  Program?  4  . 

0       Are  the  divisions  of  responsibilities  among 
and  between  PIC  staff  and  members  cliar? 

*  Does  the  PIC  have*  a  complete  statement^  of^ 
mission  and  function? \ 

#  ,      is  tHe  Council  structured  most  effectively 
and  efficiently  to  facilitate  both  decision- 
making and. program  oversight? 

PIC  Operations*   '  - 

i  Is  the  PIC  well-staffed?    Are  the  PIC  and  the 

siaff  assigned  to  or  hired  by  i£  working  pro- 
i     ductively  together? 

m       Is  there  an  active  core  of  €he  PIC •  membership 
large  enough  to  sustain  continuous  decisions- 
making  and  program  support  activity?    Can . 
vacancies  be'  filled  in  a  timfely  fashion,  and 

•  are  they  filled  using  criteria  that  help  • 
su»t£in  and  build  this  core  of  active  members? 
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£re  PIC  relations  with  the  prime  sponsor  goqi 
.enough  to  allow  the  PIC  to  have  ahy  leverage  • 
Jon  the  whole  CETA  system?  . 

Is.  the  PIC  visibl'e  and  positively  received 
in  the  'Local  bu^irtess  community? 

% 

PIC  Procedures  and  Methods 

Does  the  PIC  have  administrative  procedures 


and  operating  guidelines  in  place?    Are  they' 
useful  and' current? 

Is  the  planning  process  for  Title  VII  orderly 
and  does  it  allow  enough . time  to  make  reasoned*) 
decisions?  . 

Is  the  planning  process* structured  to  include 
individuals  and  institutions  whose  support* 
for  the  final  product  is  important?  '  r 

Does  the  Pl6  have,  a  systematic-  method,  for 
monitoring  program  performance  and  taking' 
corrective  action  as  necessary?  V 

Does  the  PIC  have  at  systematic  plan  for 
evaluating  the  effectiveness  of  each 
activity  and  its  total  progr|m?  ' 

Is  the  evaluation  data  which  is  collected 
sufficient  in  quantity  and  quality  for  PIC 
decision-making?  % 


B.     DEFINING  PRIORITY  DEVELOPMENTAL 


V 


NEED  AREAS 


By  answering  the  foregoing  questions,  your  PLC  can 
identify  internal  "barriers"  to  effective  Pld  performance 
and  your  PIC's  developmental  rte^ds.'   Needs  which  you  might 
identify  throvwh  this  ptocess  include  the: 

•  nfced  to  reorganize  or  to  restructure1  ♦ 
the  Council  w 

•  need  to  clarify  ^taff  and/or  member 
-roles  auid  responsibilities   *  % 

•  •  need  to  improve  statfvor  Council  per- 

formance either  in  terms  of  work  output, 
,      0   .  relation's  with  one  another,  relations 
with  CETA  Prime  Sponsor,  or  external 
j  relations 
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•  need  to  develop  net/  ot  more,  detailed  *'  ' 
administrative  procedures  and"  operating 

v  guidelines  -* 

•  need  to.  upgrade  the  PIC  planning,,  \ 
management  and -evaluation  process         f  * 

.     You  can  examine  all  of  those  barriers  'ancf  needs  which 
you  have  identified  to  determine  the  extent  to  which  they 
impede  your  PIC's  performance.     Then,  based  upon  that 
assessment,  you  can  create  a  priority  ranking  of  *he  develop- 
mental needs  of  your  PIC.     U^ing  'that ^Tanking,  you  can  then 
select  those  needs  wfeich  you  are  going  to  address  in  terms 
of  instituting  changes  rwithirf  this  fiscal  yeax.     A  process  ' 
you  might  employ  to  accomplish  this  priority  needs  ranking 
is  described  below: 

1 

1.     Develop  an  organizational  and  operational 
•    self-assessment  questionnaire,  using 

questions  similar  to  those  presented  <  % 

earlier,  to  be  completed  by  all  Council 


members. 

V  2.     H^e^Council  members  complete  the  ques- 
tionnaire .independently.  *  < 
.%          *                          »  *■ 

3.  .  Tabulate  response^  of  Council  members  t 
in  order  to  determine  a  ranking  of  • 
priority 'needs.  a  ^ 

4.  Have  a  Council  meeting 

-  to  examine  results  of  ques- 
tionnaire ranking; 

•.to  determine  whether  priority 
«  order  should »be  changed  or  * 

i  other  needs  shpuXd  be ' included; 

-  to  decide  what  needs  should 
be  addressed  in' this  fiscal     •  m 
year.  .        .  '  \ 

* 

5.  Create  a  PIC  ,organi?atipnal  and  operational 
developmental  agenda  based  upon,  meeting 

Completing  this*  step  enables,  your  PICT  to  begin  to 
shape  the  framework  through  which  it  will  achieve' its  pro- 
grammatic purpose  and  objectives.     Step  5  which  follows 

—  #  "*     •  , 

describes  specific  actions  vhich  your  PIC  fnight  take  to 

*■  *.  •  •  m  ' 

-^efine  its  basic  organizational  and  operational  framework. 

eric  •    „;_M  ■     / . 


>.   •  .  •  .  / 

Van  Horn,  Carl,  Evaluating  PSIP  Programs  and 
Activities:     A  Guide  for  PIC  Members  and. 
Staff   (Eagleton  Institute;   Rutgers,  the State 
University  of  New  Jersey,  1980). 
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*te>  3      \  Establish  Initial  Need  Priorities 


The  purpose  of  this  step  is  to  take  the  data  pbtained  by  :ft>ur  > 
PIC  in  "Stage  A.-    Assess  Local  Needs"  and  review  it  to  estab- 
lish a  preliminary  ranking  of  programmatic  needs  and  priorities, 
Establishing  these  priorities  involves: 

,  .  defining! employment  generating  needs;  ~ 

.  selqcting^target  groups  to  receive  program 
ser>Aces; 

.  *  determining  occupations  for  training  pro-      •  , 

grams;^  and 

.  defining  training  and  development  service^ 
needs.  *  y 


c 


Q  Key  Concepts 

Employment  Generating  Needs  -  local  needs  related  to 
developing*  activities  or  services  to  create  jobs  or  to  y 
otherwise  improve  employment  opportunities  for  CETAr Eligible  • 
participants.  f 

Target  Groups  -  those  significant  segments  (e.g.,  youth,  blacks, 
welfare  recipients,  displaced  homemakers)  of  the  local-  popula- 
tion who  are  selected  as  the  foci  for  PIC's  arograitpning  efforts. 

Occupational  Training  Priorities  -  local  "grcfwth"  occupations 
selected  as  those  for  which  PIC  should  cjeve^op  training  programs 

Participant  Service  Needs  -  training  ahd  ^development  assistance 
needs  of  the  individuals  in  those  groups  which  have  beer?     •  , 
selected  as  targets  for  PIC's  programs.' 


fl(  1(  Knowledge/Captbintlcs  Rttlng 

Review /the  information  provided  above,  f*Rate  your  PIC's  knowledge  &nd 
capabilities  with  referteoc^  to  this  step  on  the  scales  below. 

•  Knowledge 

•  Capabilities 


1  

1 

1 

i  ' 

•  J 

Excel-' 

Jfcbove 

Aver- 

Below 

.»  Poor 

lent  . 

Avg. 

age 

Avg. 

\ 

I 

 1  

i 

1 

Excel- 

.. Above 

*  Aver- 

Below 

Poor 

lent 

'  Avg. 

age 

Avg.% 

Based -upon  your  ratings,  decide  whethe^  you  need  V?  read  the  content 
information  on  this  step  which  follows  im^liately  or  proceed  to  the 
next  step.  V 
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PME*  PROCESS,.  NOTES/ACTION  PLANNING  PAGE 
NOTKS  (Points  that  a^e^Jjnportant  which  you  want  to  remember) 


K  % 


r 


QUESTIONS   (Items  on  which  you  are  unclear  or  want  to  get  additional 
information) 


ACTIONS   (Specific  steps  that  you  intend  to ^take  to  implement  your 
s  PIC's  PME  process  in  this  area) 

Steps  Resources/Persons  y  Time  Frante; 


/ 


J* 
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/ 


^      -    f  INTRODUCTION 
As  you  know,  there  are  two  broad  categories  of  programmatic 


cnl 

'  rt T 


activities  which  your  PIC *ean  support: 


y»ur  PIC*can  suppc 

g  yfifl  Development 
s  Bpecif ically^  dii 


Training  jtj^g  Development  Projects  - 
projects  specifically*  directed  tdward 
CE^FA  participants  to  prepare  them  to 
sedure  unkubsidized  employment, 

•  Employment  Generating  Services  - 
projects  or  services  not  specifically 
directed  at  clients,  intended  and  de- 

*  signed  to  generate  increased  employment 
opportunities  in  the  private  sector  for 
the  CETA  participant. 


\ 


In  this  step,  your  PIC  establishes  preliminary  priorities 

•  >  h  . 

fox;  both  of  these  categories.    This  involves: 

#     defining  employment  generating  ser- 
vice needs 


•     selecting  target  groups 


/ 


4> 


determining  occupations  for  training 
programs 

defining  participant  training  and 
development  service  needs. 


\ 


A.     DEFINING  EMPLOYMENT  "GENERATING 
SERVICE  NEEDS 


Under  the  Employment  Generating  Services  heading, 

your  PIC  has  the  responsibility  to:  - 

>■  /\ 

•  attempt  to  bridge  the  gap  between^ 
the  private  and  public  sectors 

•  help  to  create  new  job  opportune-  -  \ 
ties  for  the  un-  and  urider  employed 

•  develop  innovative  alternative*  to 
traditional  approa<fhes  to  meeting 
employment  and  training  needs. 
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Asking  the  Right  Questions  ^ 

To  »discharcfe  these  "respohsibilities,  your  PIC  needs  to- 
evaluate  local  circumstances  ancL  conditions.     By  using  your 
needs  assessment  data  to  answer  questions  ^such  as  the  following, 
you  can  determine  which  needs  are  most'  important  $t  this  time. 

Is  there  a^lack  of  awareness  of  -PSIP?- 

•  Is*  there  a  negative  impfessioft  toward  , 
governmental  employment  and  training  pro- 

;        grams?         • "  * 

•  Do  good  job  Opportunities  exist  within  the  S 
local  environment  which  gcTunfilled  because  *** 
of  lack  of  contact" between  governmental ly-  . 

>  sponsored  programs  and  local  business  estab- 

I        lishments?  .  *s 

Is  there  a*  need  to  h^lp^qreate  jobs  or  to 
develop  th£  economic  climate  of  the  com^ 
^munlty?  *  * 

*  •  Is  there  effective  cooperation  and  communica- 
tion among  those  agencies  which  qonstitute 
your  local  employment  and  training  network?  « 
«  * 

•  HaVe  governmentally^ sponsored  programs  been 
designed  to  be  directly  responsive  to  the 
needs  of  local  businesses? 

•  What  do  local  employers  feel  inhibits  their 
employm^pt  of  tfye  structurally  unemployed?^ t 

•  What  services  can  be  provided  to  enhance  the  • 
local  business  communities  receptivity  toward 

your  programs? 


Identifying  Priorities 

.Your  answers  \o  the  foregoing  questions  will  identify  the 
areas  in  which  there  ajoe  high  needs  and  help  to  determine  the 
design  of  the  Employment  Generating • Services  portion  of  your 
plan.     Possible  general  need  areas  and  the  types  of  needs 
within  thne  areas  which  might  be  identified  through  this 

'  r  < 
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process  are  listed  below; 

Economic/Community  ]  Development 

•  Tc^  enpourage  local  business  to  remain  in 
community.  -*<-'. 

*  To  provide"  local  bdsiness  Vith  funding/. 

f  assistance^for  expansion*  * 

'  #       To  attract  new  businesses  -  (foreign  and 
domestic)  to  community. 

#  Ta  forjn  pew  small  businesses  which  will 
employ  clients  as  proprietors  or  >as  em-  j  ^ 
ployees:                                $  "  . 

4.  •       To  encourage  federal ipr.funded  redevelopment 
*  projects  which  require  jobs  for  targeted ^  > 

disadvantaged  c 

•  To  brfng  workers  and.  employers  together 
through  co^unitY  infrastructure  improvement."  * 

Linkages  - 

#  To  improve  operational  relationships  with  / 
,  *    ^oGFther  local    state  or  federal  government  ageix^egj 

involved  in  job  t(^ation  ahd  econcxnic  development. 

,  #       To  improve  coordination  training/ job 

*  development  efforts  pf  PIC  and  other  com- 

i      munity  and  business  groups .  '  •  ^ 

*   

•  To  combine  resources  of  PIC  and  local 
labor  unions  to  provide  OJT.  1 

«       To  bring  schools  and  bus'fcness  together* 
to  better  match  school  progr^ns  and 
business  -needs.  •  * 


Marketing  .  % 

.#       To  increase  employment  of  clients  and 

others  through  use  of  tax  credit  incentive. 

•  To  encourage  employer  use  of  PIC  projects/ 
clients*     *  .  / 

#»      To  j^quaint  .qmploye^s  wifeMjfcnge  of  VIC/% 
4         CETA  programs  and  services?'  * 

•  To  aid  employees  in  making  best  use  of, 
available  PIC  projects/CETA  clients.  • 


\      Wc^et/Workplacfe  /*y' 

#       Tq  fill  local  employer  needs  by  recruiting. 
=>  and  training^ individuals  who  have  diffi- 

culty' joining  labor  force. 
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To  give  clients  needed  skills  in  job 
hunting  and  work  related  behavior  so  they 
will  be  acceptable  to  local  employers  who  • 
have  openings'. 

To  match  available  clients  with  identified 
needs .of  local  employers  to  create  specific* 
jobs  for  CETA^eli^yLble  participants , in  local 
businesses.  ~      %  * 


Lattor  Market  Analysis 

•        To  *secupe  better  information  on  local  labor 
market  and  business  conditions. 

Tavma£e  the  decisioTf*of  which  needs  are  most  important, 
you  could  have  your  Council  first  identify  all  those  needs 
which  exist, /and  ther\  vote  to  determine  a  priority  ranking 
among  frt^se  needs.    Your  PIC  could  then  decide  which  needs 
should  be  priorities  for  this  fiscal  year. 


B.     SELECTING  TARGET  GROUPS 

To  initiate  the  process  of  -priority  setting  on  the 

participant  or  training  and  development  side,  your  PIC  should 

select  the  groups  .who  will  be  your  targets  'for  both  'training 

and  develqpment  project  and  services.    ¥ou  should  do  this  by 
« 

taking  the  data  secured  in  your  needs  assessment  into  account 
The  decision  of  whom  tg  serve  i^  very  much  a  local  option  and 
will-  be  dependent  on  community  needs. 

To  assist  your  PIC  in  making  its  decisions  regarding 
target  groups you  might  wajit  to  establish  certain  factors  r 
or  considerations  to  look- at  in  making  your  determinations. 
Possible  factors  could  be: 


7 


relative  overall  severity  of  need 
of  each  participant  grodp 

"employability"  based  upon  ecjucation, 
length  of  unemployment  of  members  of 
participant  group 
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percentage  of  participant  group  being 
served  by  other  employment  and  training 
programs  ^. 

particular  or  special  needs  of  sub- 
groups (e.g. ,  displaced  homemak^rs; 
ex-offenders;  alcoholics;  Vietnam  . 
vets) . 


Establishing  Priorities 

The  pages  which  follow  describe  an* experimental  method 
which  one  CETA  Prime  Sponsor,  employed  to  select  target 
groups.     It  is  presented*  here  not  as  'an  absolute  way  for 
correctly determining  target  groups,  but  as  one  method 
your  PIC  might  Want  to  consider  to  u'se  or  to  modify  to 
'develop  a  process  for  target  group  selpctioii  which  Will 
work  for  you. 

c 

1 •    Modgl  Procedure 

The  objective  of  this  model  procedure  was  tb  deter- > 
mine  target  group  priorities  through  a  process  whiqh  would: 

#      enable  a  quantified  assessment  of 
the  relative  needs  of  target  groups 

0     allocate  funds  in  relationship  to 

that  need  determination    ,  fc  ^ 

m   m     provide  for  the  snAoth  resolution 
of  internal  planning  and  Council 
conflicts 

assure  that  decisions  were  reached 
within  a  specified  time  frame. 


In\i 


Injthis  method,  a  tatget  group1 s  "priority?  was  defined 
as  the  group *s  proportion  in  the  "universe  of  need"  (all 
eligible  CETA  participants),  in  relation  'to  the  final  funding 
proportion  whic^  it  w^s  to  receive  from  the  total  amount  of 
CETA  funds  to  be  expended.  ,  High  priority  groups  would  receave 
funds  in  a  proporti6n  greater  than  their  proportionate  . 
presence  in  the  total  target  group  population  for  the  CETA 
program.  • 
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, The  general  process  which  was  followed  to  app£y  this 
concept  .was  as  follows:        .     *  + 

a*    "The  Council  members  were  oriented** 
*  -ttftfche  process  of  determining  target  » 

.  .  <  group  priorities* 

.  ^  *»  l>.     The  staff  developed  the  following  v 
information  and  provided  it  to  th$ 
members:  •  1 

/.  ,  -  Table  displaying  distribution 

of  target  group  categories  by 
v   '     *'v  age  and  race 

-  -  Manpower's  funding  proportipn  0% 

v    ;  choices  in  the  previous  year 

%x]  for  each  target  group  { 

-  Issue  papers  under'the  titles: 

x*  mt  Handicapped  persons 

f  persons  on  welfare 
0  Spanish-speaking  population 

#  Veterans  0 
.  #  Blacks 

•  Unemployed  workers  45  years 
.of  age  or  olden 

c.     Based- upon  a  review  of  this  informa- 
'  tion,  a  numerical  weighting  sfcheme 

consisting  of *  the  following  scales 
was  applied,  to  each  target  group 
•  <  category:  X. 

heist  Employ.  1  2  3  4  5  6  7,  Most  Employable  ^ 

Least  Obliga.  12  3  4  5  6^7  A  Most  Obligations  \ 

*                    '  In  School  1  2  3  4  5  67    Out  of  School  _ 

,  ^                  Non-Veterans  .1  2  3  4  5.6  7  Veterans 

' _  t      Handicapped  1  2  3  4  5  6  7  m  Non-r  Handicapped 

1     9    ti.     The  results  o£sthis  weighting^by  target 
group*  category  were  then  revieVed  by  the* 
Council  in  order  to  determine  relative 
^  degrees  of  need  and  priorities  for  each  category. 

^  e.     This  data  was  tjien  used  to  assign  fusing 

5.   percentages  to  each  target  .group.  « 
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.  Again,  this  process' is  hot  presented  as  the  final  word 
on  target  ^roup  selection,  but  as  one  attempt  to^help 
raake^  the^  selectign  of  target  groups  more  rational  and 
comprehensive.     Np  matter  which  target  groups  your  . PIC 
selects,  however,  you  will  need  to  determine  what  training 
and  development  services  and  projects  to  undertake  to 
assist  them.  ■ 


C.     DETERMINING  OCCUPATIONS  FQR 

TRAINING  PROGRAMS 

.  v   

On  the  training  and  development  side  of  your  PIC's 
program  ledger,  ^pu  should  determine,  from  among  al\  of  ■ 
those,  "growth"  occupations   (due  to  job  turnover  or  expan- 
sion)    as  revealed  by  your  labor  market  analysis,  which 
should  be  priorities  for  training  programs  to  be  sponsored 
through  your  PIC. 

To  make  this  determination,  *y°u  should  establish  some 
criteria  that  you'  can  use  to  review  the  growth  occupations 
and  their  "suitability"  for  the  participants  in  your  program 
Possible  factors  for  which  your  PIC  could  establish  specific 
criteria  ^nclude: 

.    *    •        total  number  of  projected  openings 
for  -the  year 

•  percentage  of  "entry-level"  openings 

\  /v  *  ^  '  projected  growth  rate -for  a  5-year  * 
^      *    ^period  ,  . 

^     hourly  st'arting  salary 

•  promotional  potential 

•  twrture  offsworking  environment  and 
conditions/  • 

I 

•  ,   skill  Qr  training  requirements. 
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Steps  for  Selecting  graining  Occupations 

Steps  which  yotx  might  'follow  to  select  those  occupa- 
tions which  should  be  priorities  for  your  training  programs 
are  listed  below: 


i. 

-  f 

3. 

> 

4. 


Review  "growth"  occupations  as  revealed 
by  your  PIC's  labor  market,  analysis. 

Remove  occupations  with  highest  and 
lowest  skill  requirements. 

Eliminate  bccupations  not  meeting 
acceptable  level  of  wages,  career  poten- 
tial, stability  and  working  conditions. 

Rate  occupations  with  respect  to  size, 
growth,  current  placement  potential  and 
location.  *  * 

a.  Remove  occupations  which  have, 
conclusively  poor  pre-  and  % 
post  training  records. 

b.  Profile  remaining  occupations 
as  to-entry  requirements, 
training  content,  and  estimate 
of  tfaining  cost. 

Recommend  occupations  according  to 
target  group  priorities,  anticipated 
economic  conditions,  cost/earnings  con- 
siderations^ 

«  *   

6.     Select  occupations  on  the  basis  of  existing 
facilities,  balanced  mix  of  occupations, 
an<jl  placement  potential  as  it  appears 
at  the  time  the  list  is  prepared.  m 

for  the  most  part,  the  foregoing  steps  are  self-explanatory, 
However,  steps  5  and  6  requfre  some  further  elaboration/ 

?  » ■  ' 

Recommending  Occupations:     Once  your  PIC  gets 
to  the  point'  of  recommending  occupations,  it 
is  essential  to  consider  both  the  general 
needs  and  capabilities  of  your  target  group 
and  the-  potential  of  the  jobs  themselves.  In 
this  regard,  you  should  consider  the  age,  edu- 
cation, wofck  Experience,  and  other  important 
'defining  characteristics  o-f  your  target  groups. 
You  sjiould  also  Jtake  a  clos^  second  look  at 
the  placement  oqtlook  for  the  period  when 
trainees  will  be* completing  programs.  Then,; 


5. 


t 
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you  should  eliminate  any  6ccupations*not  suitable 
for  the  target  groups  with  priority  or  having 
uncertain  placement  potential .  ** 
• 

One  way  of  selecting  among  those  occupations 
which  remain  is  to  compare  expected  training 
costs  with  benefits  as  measured  by  expected 
wages  after  placement.    Based  upon  this 
'comparison,  you  could  eliminate  occupations 
with  costs* excessively  high  £n  relation  to 
expected  earnings.     If  possible,  you  should, 
develop  standards4with  cutoff  ratioa  in 
advance  to  help  you  do  this;,.  * 

Selecting  Occupations:    Once  yopr  PIC  has 
reduced  occupations  -  to  those  which  appear 
to  be  "best"  for  training*,  you  should  subject 
those occupations  to  a  final  round  of  con^ 
sideration  related  to  the  capabilities tof 
current  tra,inin£  facilities  and  the  necessity 
of  providing  training  that  will  appeal  to  ci 
range  of  participant. interests  and  abilities.  . 

M  - 

Indeed,  planning  your  PIC's  program  to  make* 
it' most  responsive  to  changing  conditions 
may  very  well  require  the  discarding  of  some  % 
training  facilities-,  the  acquisition  of 
others,  and  the  upgrading  of  others*    You  . 
should  make  thoSe  determinations  and  then  *' 
select  those  occupations  which  are  most-  \^ 
appropriate  to  ^>ur  situation. 


'  D.     DEFINING  PARTICIPANT  SERVICE  NEEDS 

In  addition  to  determining  the  occupations  which 
should  be  priorities  for  training  programs  to  be  sponsored 
through  your  PIC,  at  this  point,  you  should  also  identify 
the  priority  service1  needs  of  those  individuals  who, will 
constitute  the  population  for  yoyr  PIC's  program.  In^ 
certain  instances,  participants  ma^  require  assistance  or 

support  in  addition  to  training,  such  a 3: 

*■ 

,  health  care  and*  medical  services 
child!  care  t  ' 
transportation  * 
temporary  shelter 

financial  counseling  and  ^ssistahce. 
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In  others,  you  may  find  that  target  group  participants  do  not 

require  occupational  training, ' but  just  services  such  as: 

*  « 

•  orientation  to  world  of  work 

•  counseling  (employment-related  and  testing) 
•  •      employability  assessment 

•  '    job  development* 

•  jbb  search  "assistance 

•  job  referral-  and  placement 

•  Target  Jobs  Tax  Credit  eligibility 

determination 

•  vocational  exploration ' programs . 

*  ' 

You  should  use  your  needs  assessment  data  on  the  barriers 
affecting  target  group  members  to  determine  what  needs  other 
than  or  in  addition  to  training  your  target  group  partici- 
pants have.     A  simple  format  which  your  PIC  might  employ  to 
make  th^fce  determinations  is  presented  below. 


Target  Group 

Major  Barriers 

Service  Needs 

*                                                      1  '« 

4- 

The  results  of  this  assessment  will  enable  yotf  to  deter- 
mine the  service  needs  you  want  to  address 'through  your  PIC's 
program.  *  *  * 


v 


CONCLUSION 


In  conclusion,  once  your  PIC  has  defined  its  potential 
priority  employment  generating  Services,  target  groups,  occupa 
tions  for  training  programs,  and  participant^  service 
needs,  then  you  have  the  basis  for  setting  your  PIC's  pro- 
grammatic goals. 
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Manpower  Administration  (now  ETA) ,  Program 
Assessment  Guide  (Washington,  D.C,:  U.S.  • 
Department  Of  Labor,  April,  1974) . 

Mayor's  Office  of  Manpower,  City  of  Chicago, 
Planning  and  Evaluation'  under  CETA  (Wash- 
ington, D:C:  U.S.  Department  of  Labor, 
January  1976) .  m 

U.S.  Department  of  Labor,  Jobs  in  .the  Private  Sector.:  use  of 
Labor  Market  Information  (Washington,  D.C, 


Printing  Office,  October,  1980). 


U<S.  Government 


J 


0 


sup  4 
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.  Redefine  PIC  Missiort  and  Function  x^ 

*  V 

The  purpose  af  this  step  is  to K reexamine  your  PIC's  present 
definition  of  its  mission  in  light  of  your  assessments  of 
community  needs,  PIC  programmatic  arid  operational  performance, 
and  your  initial,  need  priorities  to  determine  whether  they 
are  still  appropriate  or  need  to  be  modified.    This  step 
involves^clarifying  PIC  purpose,  roles,  and  functions. 


\  PIC  'Mission  and  Function  -  basic  purr^'se,  roles  and  f esponsi- 
bilities  to  be  accomplished  by  PIC. 

•  PIC  Purpose  -  'statement  *of  ultimate  intent  and  results  expected 
xn  the  long  term  by  PIC.  r 

•  PIC  Role(s)  -  primary  approach (e£)-  to  performing  and  dig- 
charging  PIC  mission  and  function.  ,  w 

9  Advisor  Role^<g»  reviewer  and  consultant  to  CETA  prime  sponsor. 

•  Broker  Role  -  facilitator  for  connections  between  and  among 
CETA  program  and  others.  -  % 

•  Operator  Role  -  designer  andu provider  of  direct  services  either 
to  participants/  employers,,  or 'other  agencies. 


1(   Knowledge/Capabilities  Ratln? 


Review  the  informati6n  provided  above.  Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  fchifc  step  on  the  scales  below. 


•  Knowledge    ^  ^ 

•  -Capabilities 


J 


Excel- 
lent-' 


Excel- 
lent 


Above 
Avg. 

1± 


Above 
Avg. 


Aver- 
-  age  I 


Aver- 
age 


Below 
•  Avg. 

A. 


Poor 


Below 
Avg. 


Poor 


Based  upon  your  ratings,  decide  whethej  you  need. to  read  the  content 
information  on  this  step  which  follows  immediately  on  proceed  to  the 
next  step.  'm 
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PME  PROCESS  NOTES/ ACTION  PLANNING  PAGE 


NOTES  (Points  that  are  important  which  you  want , to  remember) 


\ 


QUESTIONS 


(Items  on  which  you 
information) 


i 


unclear  or  want  to  get  additional 


ACTIONS  (Specific  steps  that  you  intend  to  take  to  implement  your 
-   PIC's  PME  process  in  this  area) 

Steps  Resources/Pgr sons  Time- Frame 


r 
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INTRODUCTION 


The  CETA  Regulations  spell  out  the  following  broad  pur- 
poses and  roles  f^r  a  Private  Industry  Council: 

Purposes :     To  increase  the  involvement  of  the 
business  community,  including  small 
^         business,  minority  business  enter- 
prises, and  labor  organizations  in 
employment  and  training  activities. 


To  increase  private' sector ^employ- 
ment opportunities  for  economically 
disadvantaged  persons. 


v         Roles:  The  PIC  shall: 

•  serve  as  an  intermediary  to  assist 
the  local  employment  and  training 
structure  toJfecome  more  responsive 
to  the  business  community. 

( 

•  serve  as  \he  bus^hess  and  industry 
contact  poin£  irf  the  local  employ- 
ment and  training  system,  to  present 
the  private  sector's  view  and  recom- 
mendations for  making  programs  more 
rrfSponsi^  to  local  employment  needs. 

•  advise  and  provide  direction  tp  the 
local  employment. and  training  system 

%  on  ways  to  increase  private  sector 

jobjp^cements  for  eligible  persons.  ' 

Within  this  broad  corrtext  it  is  up  to  your  PIC  to  define  its 
specific  mission  (parpo^e  and  roles)  and  functions  more  pre- 
cisely.    The  pjrotfess  of  defining  purpose,  role,  and  functions 
is  iterative,  beginning  with  the  definition  of  purpose  and  then 
proceeding  through  functions.     The  discussion  which  follows 
highlights  general  considerations  related  to  this  definitional 
process,  and  then  suggests  a  procedure  which  your  P£C  might 
employ  to  redefipe  its  mission  and  functions. 
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LOOKING  AT  MISSION  AND  FUNCTIONS 

T^iere  is  a  lid§  range,  of  options  available  .-to  your  PIC 
terms  of. purpose,  roles,  and  functions/  ^  * 


Purpose  x  • 

A  purpose  is  a  statement  of  ultimate  intent  and  r 
expected  in  the  long  term  by  the  PIC.    Your  PIC's  purpose 
should  be  drawn  from  the  Title  VI»B"egulations  and  legisla- 
.  tion  and  tfhen  adapted  to  express  your  local  emphasis. 

Although  the  Title  VII  regulations  provide  the  initial 
mandate  for  your  PIC,  you  should  look  at  the  employment  and 
training  needs  within  the  local  community  4nd  decide  to 
broaden  or  restrict  your  scope  and  purview  to  address  those 
specific  needs. 


1 .     Sample  Statement  of  Purpose 
An  example  of  one  PLC's  statement  of  purpose  follows: 

•  To  increase  substantially  the  involvement 
of  the  private  aector  in  publicly^ financed 
employment .and  training  programs.  , 

•  To  strengthen  the  capacity  of  the  employ- 
e-went and  graining  system  to  respond  .to 

the  needs  of  both  industry  and  the  eco- 
nomically disadvantaged. 

* 

•  To  increase  the  access  to  good  jobs  for 
the  economically  disadvantaged  and 
structurally  unemployed. 

•  To  improve  the  local  investment  climate  •  y 
and  to  increase  coordination  between 
employment  and  training  and  economic 
development  programs^ 
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2.     Principles  for  Statement  of  Purpose, 

In  developing  ytfur-PIC's  statement  pf  purpose,  you 
'should  be  aware  of  the  following  principles: 

•  A  statement  of  purpose  should  be  as 
clear  and  specific  as  possible; 

•  There  should  be  a  distinct  statement  > 
for  each  priority  intent  or  result 
desired  by  the  PIC.  * 

•  The  PIC'.s  purpose  needs  to  be  uni- 
formly understood  and  embraced  by  aH 
Council  members. 

•  The  PIC's  purpose  should  be  communi- 
cated clearly  to  the  public  and  private  V 
sector. 

•  The  purpose  should  be  changed  over  time  m 
as  local  conditions  and  needs  change.  •  * 

On&e  you  have  agreed' upon  your  PIC1 s  statement  of  purpose, 
that  will  help  to  determine  the  role(s)  that  you^want  to 
play,  and  the  specific  functions  pr  responsibilities -that  * 
you  want  to  assume. 


Rble(s) 

 - — -  M 

/    1  %    There  is  no  single  prescribed  set  of  roles  for  a  Private 

industry  Council. 
*  * 

1 .     Role  Options 

The  three  basic  potential  roles  for  a  PIC,  none  of  which 

»  •  > 

exclude^  either  of  the  others,  are: 

•  Advisor  %    -  reviewer  an<J  consultant  to 
gLocal  CETA  Prime  Sponsor  in 
program  planning,  implemen- 
* t  tation,  and  review  process. 

.       Broker    %    -  facilitator  for  connections 

between  and  among  employers, 
Prime  Sponsor,  other  providers 
•  ■  of  employment  and  training 

Services,  government  agencies,  # 
and  the  community.  £ 
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•  Operator  -  designer  and  provider  of  direct 
services  either  to  participants, 
employers,  or  other  agencies. 

As  suggested,,  the  CETA  ReguleuiiOTis  require  -  your 9  PIC  to 
-  play,  at  lea^fc^the  advisor  pr  cprtfultant  role  in  certain 
^treas.    Your  responsibility ,  at  this  point,  is  to  clarify  the 
additional  role(^)  that  your  Council  wants  to  assume,  if  anyy 
with  reference  to  the  PS IP  and  to  your  local  employment  and  ^ 
training  system  generally.*  w  ♦ 

No  one  tole  or ,  set  .p^^bles  is- inherently  superior  -to 
another.    «The* important  factor  fpr  your  PIC  to  cpnsider  at 
this  juncjbuz;e  is^ija*t  is  needed  and  what  will  work  within  the 
loqal  enj/ironment.    Your  overriding  responsibility  is  to 
ensure^that  your  role  decisions;  reflect  the  private  sector's 
interests  and  contribute  to  a  more -effective  employment  4 
training  delivery  system.  •  m  * 


2.     Factors  to 


idfer 


Faators  which 'you  should  take  into  account  *in  making 
'cleaisionfe  about  what^role(s)  your^PIG*  wants*  to  play  include: 

-•  limitations  imposed  by  the  amount  of  / 
available  resources  rf  v 

•  the  "perceived"  status  and^bapabilities 
of  the  lofcal  CETA  system.  Positive 

*, status  a n4  strong  capabilities  plight 
lead  you*  to  JtSiime  an  advisory  role. 
•Lesser  status  and  capabilities  may 
irfdicate  the  need*  for*  more  extensive 
involvement* 

•  •  •  , 

•  .local  economic  conditions.     Poor  eco-* * 
ftomic  .conditions  may  suggest  more 
active  involvement. 

•  present  strength  and  productiveness  of 
private  sector  ties 


•  l4>cal  political  condition's* 
i   #• -nature  of  CETA-PIC  relations 


A 


/ 


•* jurisdictional  status  of  the  PIC*  A 
variety  of  roles  may  be  more  difficult 
to  carry  out  in  multi jurisdictional 
areas 


•  nature  and  level  of  PIC  , staff ing. 


// Based  upon. the  role(s)  thht  your  Council  selects,  you 
should  determine  the  functions/responsibilities  that  you  p 
wish  to  perform.  *% 

,  4    *  *    1.    Basic  Responsibilities  .» 

A.  list  of  Mandated  jmd -other  basic  responsibilities 
fdr  your  PIC  is  presented  .below:         '  .  ' 

*  /  «,    •  To  prepare  a  plan  for  the  conduct  of  t 
Title  VII  activities  and  programs  in 
.  conjunction1 with  the^Prime  Sponsor 

•  To  participate  #oUly  in  the  formula- 
tion and' development  of  the  Prime  \ 
Sponsor's  plan  for  conduct  of  all  %  ■ 
other  CETA  programs 

•  -To  participate  in  and  "implement  ^activ- 
ities authorized  under ,PSIP  in  coo^era- 

.*    tion  Vitlh  the  .Pxime  Sponsor. 

~  <     *     *  *  •  ' 

•  To  set  priorities- to  meet  local  business 
conditions  ,  . 

•  To  become  a  part  of  the  overall* governance 
structure  of .the  entire  prime  sponsorship 

•  To  provide  genuine  # decision-making 
authority  as  representatives  of  the 
private  sector 


•  To  promote  development  of  specialized 
labor  market  information  upon  whfch  the 
PIC  may  make  productive  decisions  and. 
we]#l-t£ought-out  plans 

•  To  offer'  large  and  smalL  businesses 
•opportunities  to  exert  their  leadership 

in  PlC-^^olicy  and  programs 


xxx-m 


•  To  sustain  continuous  decision-making 

and  .program  support  activity  for  \ 

private  sector  initiatives 

«>  * 

-> 

•  To  stimulate  more  comprehensive  planning 
between  the  PIC  and  Prime  Sponsor  with 
more  responsiveness  to  private  Sector 
needs 


^    •  2..,  Other  Functions 

— — — — /  • 

Example's  of  possible  'discretionary  functions  as  they 

relate  to  the  three  basic, roles^identif ied  earlier  include 

Advisor 

•  Identify  factors  which  contribute  to 
program  effectiveness.. 

to  Examine  the  performance  of  CETA  and  the 
Employment  Service. 

•  Review  and  comment  on  plans  of  economic 
development  agencies.* 

*  *  * 

Broker  ^ 

*  % 
Between  individual  employers  and  publicly-funded 
programs: 

•  Disseminate  PJG* program  information 
to  employers.' * 

•  Di'sseminate  information  oti  the  Targeted 
Jobs  Tax  Credit  and  other  incehtive-type 
programs.  * 

•  Develop  and  market  model  contracts. 

•  Provide  assistance  to  employers,  to  re- 
duce the  Administrative  burden  of  par- 
ticipating* in  employment  and  training 
progr'ams . 

•  Encourage  employers  to  develop,  and  to 
coordinate  with  training  agencies ,  job 


.skill  requirement  forecasts. 
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Between  employers  (as  a  group)  and  other  groups 
"or  institutions  in  the  community: 

•  Coordinate  Title  VII  programs  wijth 
other  employment. and  training  activ- 
ities.  %/  s* 

•  Develop  linkages  between  school  and 
work . ,  * 

•  Develdp  linkages  between  employment 

.  '  ,  an4  training  and  economic  development 
programs. 


Operator 

Programs  .for.  workers:  *.  . 

•  Provide*  wqAd^bf  wprk  training. 
Provide*  vocatioaal  sJAl Is  .training.        '  • 

•  Provide  training  and  development  and 
support  service*/ 

>^  •  Prcj*ri4e  entreprfcneurship  training.  * 

Programs  ;-f6r  emplpS^^r^s^ 

-  •  Provide  s^cfip^  linked  to  CETA 
c  0       tr^^^^tfntrplac«Re^nt.        w  ^ 

•  Pr<$^^  development  and 
job  j^eat ion ^tfelated  services. 

Programs  fQ&  education/employment  and  training 
*  staff:         .  \  *  *  * 

•  Promote  exposjnre  >to  prij^te  industry. 

t  Provide  as^isMtnce^n  accomplishing , 
spepific  tasks. 

/    %     \*    •  '  / 

•AgaiA,  tfs*yitfv  r^lfes,  there  is  no  preferred  set  of  func- 
tions ox  :respo»3ibil ities.    The  essential  considerations 
should  be  that  .(-1)  your  PIC  performs  those  functions  and 
assume* those  r^pponsibil'ities  required  given  local  conditions 
and'  needs,  and  (2)  that  your  functions  flow  from  and  relate 
dirSjStly  "back  to  ^our  definition  of  purpose  and  roles. 
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B.   ^MAKING  DECISIONS 

« 

Your  PJG  should  employ  whatever  process  you  feel  Will  be 
most  useful  and  expeditious  to  make  its'  decisions  regarding 
mission  and  functions^.     A  process  which  one  PIC  employe^  to 
make  these  dej^sions  ijt  outlined  below: 

a.  The  staff  developed  an  information  paper  m 
entitled  Roljes  of  the  Private  Industry 
Council, o  This  paper  summarized  the  op- 

taons  available  to  the  PIC  and  described 
activities^-which  micjht  be  undertaken  in 
each  role.v  * 

b.  This  paper  wag  distributed  to  the  PIC 
members . 

c.  At meeting  was  convened  to  discuss  goals 
and  activities. 

d.  At  the  meeting,  a  summary  outline  of  the  ^ 
information  paper  was  distributed,  as 
were  pencils  and  paper. 

e.  Each  participating  member  was  invited  to 
wri^e  out  his/her  priorities  for  the  PIC.  * 

f .  The  priorities  were  then  read  aloud  and 
recorded  on  a  flipchart.. 

g.  A  discussion  followed  and  key  points  of 
the  discussion  were  also'  summarized  on 

the  flipchart.  »/ 

*  «  * 

h.  After  that  meeting,  all  written  responses  * « 
from  members  were  tabulated  according  to 

the  outline  of  the  information,  paper . 

i.  The  result^  from  this  tabulation  and  the 
interactidn  from  the~meeting  were  analyzed 
by  staff  to  develop  themes  including: 

-  sketch  of  PIC's  possible  long- 
term  role  ^x 

•  -  possible  functions  to  be  ^per- 
formed 

-  type  of  projects/activities  1 
to  be  supported  . 
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This  information  was  fed  back  to 
C««ncil  at  another  Meeting  and  a 
decision-was  reache^  on  the  PIC's 
mission  and  functions. 


CONCLUSION- 


In  conclusion,  redefining  your  PIC's  general  mission 
and  functipns  is  an  essential  precondition  to  establishing 
its  programmatic  orientation  and  Strategy . 


National  Alliance  of  Business,  Roles,  . 
Organization  and  Programs';  #asicfe  for  the 


Private  Industry  Council  (Washington,  D.C. 
National  Alliance  of  Business,  1981). 

Chamber  of  Commerce  of  the  United  States, 
The  New  Private  Sector  Initiative  Program 
(Washington,  D.C  :  Chamber  of  Commerce  *of 
the  U.S.,  1979.)  . 
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Refine  PIC  Organizational*  and 
Operational  Framework  ' 


The  purpose  of  this  step  is  to  reexamine  your  PIC  s' basic 
working  arrangements  to  see  if  they  are  still-  liable  and  to  * 
revise  and  refine  them  as  necessary*    Areas  to  look  £t  incltfde:' 
structure,  staffing,  operational  procedures,  and  organizational 
status. 


Structural  Issues  -  size  and  composition,  membership  selection 
processes,  and  committee  types  for  PIC.  *  «^ 

•  Staffing  Issues  -  arrangement  for,  level,  nature,  and  division 

of  labor  among  those  individuals  to  provide  staff  services  forPIC, 

•  Operational  Issues  -  Council  by-laws;  meeting  procedures;  and 
processes  for  determining  roles  and  responsibilities  of  members 
and  staff,  and  selecting  officers  and  assigning  their  duties. 

•  Organizational  Status  Issues/^  fundamental  arrangement,  (e.g. 
position  within  Prime  Sporfsor,*  incorporation,  part  of  existing 

"  local 'business  organization)   for  PIC's  operation.  * 

1  - 


g(  j(  Know! edge/ Capabilities  Rating 

Review  %he  information  provided  above..  Rate  your  PICf£  knowledge  and 
capabilities  with  reference  to  this  step,  on  the  scales  below. 


•  Knowledge 


•  capabilities 


1  

I 

1 

i 

,  J 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

•  age 

Avg. 

t 

I 

• 

1 

i 

 1 

Excel- 

Above 

Aver- 

Below 

Poor, 

lent 

Avg. 

age^ 

Avg. 

BasedjiEon  your  ratings,  ^decide  whether  you' need  to  read  tha  content 
infSrmatlbiv^on  this  step  which  follows  immediately  or  proceed  to  the 
next  step. 
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>ME  PROCESS  NOTES/ACTION  PLANNING  PAGE 


NOTES  (Points  that  are  important  which  you  want  to  remember) 


QUESTIONS   (Items  on  which  you  are ^unclear  or  want  to  get  additional 
information) 


ACTIONS  (Specific  steps  that  you  intend  to  take  to  implement  your 
PIC's  PME  process  in  this  area) 


Steps 


Re  source  s/Persoijs 


Time^  Frame 


0 

ERLC 
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INTRODUCTION' 

Your  PIC  has  enpifinous  latitude  in  deciding  on  its 
methods  of  brganizationand  operation.     In  this' step,  you  use 
the  data  gathered  and  analyzed  in  assessing  your  organiza- 
tional and  operational  performance  to  decide  whether  to 
initiate  changes.    Areas  in  which  you  might  want  to  modify 
working  arrangements  are: 

•  structure 
*  •  staffing 

1  •  operational  procedures  — 

•  organizational  status* 


-     A.     LOOKING' AT  STRUCTURE 

The  size,  composition,  and  structure  of  your  Council 
itself  should  be  dictated  by  local 'needs,  interests,  and  the 
mission  and  functions  which  ypur  PIC  establishes"  for  .itself 
.Factors  which  you  should  consider  in  reexamining  the  nature 
of  your  PiC  include: 

•  relationships  with  other  CETA  programs 
*  •  physical  requirements  for  meetings 

•  manageability  of  PIC  for  staff  and  chairpersons 

•  workload  requirements  upon  Council  members 

•  representativeness  of  the  Council- in  terms  of  t 
compoaritldn  (Are  small  and  minority  ; 
businesses  adequately  represented?  Does 

the  Council  reflect  the  largest* possible 
„  segment  of  community  concerns?) 

•  staff  and  serviced  necessary  to  support  the  PIC* 

Specific  items  which  metit  special  recons id-era t ion  under  the 

structural  heading  include  your  PIC's*  % 

*  * 

e       i  ✓ 

^         •      Membership  Selection  Process  > 
\        •      Size  and  Composition  of  the  Council 

•  •     Committee  and  Subcommittee  Structure  J 
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Membership  Selection  Process  ■ 

The  ^itl-e  VII  legislation  provides  that  as  a  PIC  is-  being 
established,  the  Prime  Sponsor  will 'appoint  a  chairperson  ar*d 
members'.     Once  the  PIC  is  established,  a  chairperson  from 
business  and  industry  is  selected.    While  it  is  tjie  responsi- 
bility of  the  chief  elected  official  to  appoint  members,  PIC 
members  are  often  involved  in  nomihating  new  members  for  the 
PIC.    Therefore,  at  this  point,  you  should  examine  your  PIC's" 
present  process  of  appointing  members  to  decide  whether  to 
alter  it.     To  th$  extent  feasible,  you  should  attempt  to  forge 
a  supportive  and  cooperative  relationship  to  co-jointly  select/ 
recommend  PIC  members  with  the  Prime  Sponsor  staff.     In  this* 
regard,  you* should  ensure  that  th^  arrangement  which  exists  now 
or  is  developed  is  one  which  benefits  Jpusijiess,  the  CETA-eligibl 

population  at  large,  the  community,  and  the  government. 

*•* 

Size  and  Composition  of  Boarfl  . 

o There  is  no  ideal  PIC  size.     Your  Council  should  be  large 
enough  to  be  truly  representative  of  the  private  sector  and  m 
include  representation  from  other  groups'  as  mandated  by  the 
Title  VII  legislation.     One  Important  point  to  remember  is 
that  an  extremeiy  large  PIC  membership  is  unwieldy  and  creates 

complex  organizational  problems. 

« 

In  terms ^pf  your  PIC's  composition,  Title  VII  legislation 
requires  that  51%  of  the  PIC  ^membership  come  from  business. 
Title  VII  legislation  also  addresses  the  participation  of  small 
and  minority  businesses  as  well  as  members  representing  labor 
unions,  community-based  organizations,  educational  institutions, 
and  economic  development  councils.    You  should  decide  the  rela- 
tive composition  of  your  PIC.  based  upon  local  ne£2ls.     Ii:  you  • 
feel  that  "business  speaks  btetter  to  business,"  you  may  want 
to  increase  private  sector  representation  to  more  than  51%* 
On  the  other  hand,  based  upon  local\circumstances,  you  may 
surmise  that. a  broader-based  representation  is  more  important.  t 

Another  important  consideration'  relatjed  to  composition  is 
the  type  of  person  that  your  PIC  wants  to  get  on  the  board. 


Options  inelude: 

•  heads  of  major  corporations  or  organiza- 
tions who  can  lend  their  prfestige  and  who 
can  influence  the  16cal  cpmrounity  as  a  * 
whole  *to  participate  in  and  support  the 
activitifes  of  the  PIC 

•  company  operational  managers  who  may  offer 
less  inf luence%but  either  greater,  technical 
expertise  or' potential  for  direct  hiring 

of  CETA-eligible* participants 

•  small  business  (500-525  persons  in  size) 
owners  who  pan  be  enlisted  to> expand  the 

*         pool  of  potential  employment  opportunities 

for  CETA-eligibie  participants  ,000000^^ 

•  *• 

•  top  level  representatives  from  governmental 
economic  development  agencies  or  similar 
private  sector  investment ^concerns  who  can 
cooperate  in  sponsoring  joint  ventures. 

" s  Factors  to  take  into  account  in  determining  the  most 
appropriate  size  and  composition  of  your  Council  include: 

•  the  size  of  the  J»abor  market  area 

•  the  nature  of  the  local  business  estab- 
lishment  the  number  and  type  of  local 
business  organisations  and  groups  ' 

•  the  nature  of  the  existing  CETA  Advisory 
Council 

•  the  staff  and  services  support^  required 

to  accomplish  PIC  activities^  '  - 

t  Regardless  of  the  final  size  and  composition  of  your 
Board,  your  PIC  should  atteppt  to  have  or  secure  individuals 
in  its  membership  who  have: 

•  •     a  conviction  about  the  purpose'and 

functions  of  the  PIC 

%       a  commitment  to  the  mission  of  the  PIC 

x  a  willingness  to  devote  time  and  effort, 

to  the  PIC. 
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Committee  and  Subcommittee  Structure 

■  p  

Tfie  structure  normally  used  to  facilitate  Council  opera- 
tions is  the  use  of  the  committees.     Standard  committees 

within  PICS  normally  include:  *  / 

». 

_  •  4    Executive  committees 

•  Standing  committees    \  *  / 

•  Ad  hoc  committees 


1.    Types  of  Committees  -  k 

Executive  Committer:    Executive  committees  are  usually 
composed  of, the  Council  officers,  the  chairpersons  of  the 
standing  committees  and  other  members  as  prescribed  in  the 
by-laws.     The  chief  fyinctlon  of  the  executive*  committee  is 
to  plan  the  work  of  the  Counqil  and  to  exact  responsibility 
by  providing  agenda  and  other  guidelines  for  action.  Another 
of  tue  executive  committees  roles  is  to  provide  guidance  to 
the  staff  between  Council  meetings.     Executive  committees 
also  normally  have  authority  to  make  important  decisions  in 
emergency  situations. 

Standing  Committees:  Standing  committees  are  established 
on-*  permanent  basis  and  may  be  described  in  the  PIC's  by-laws 
Optional  ways  for  your  PIC  to  organize  and  structure  its  *"  ^ 
standing  committees  are: 

'  •  Around  program  activities  such  as: 

-  OJT/skills  training 

-  pre-employment  training 
•  marketing* 

4 

•  Around  industry  groupings  such  as: 

-  construction 

-  manufacturing 

-  transportation 

•  Around  operational  or  functional  areas  such  as: 

-  planning  / 

•  -  monitoring  and  evaluation 

•  7  finance 

-  public  relations. 

-  personnel* 

III-122      '  f 
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1  #       Ad  Hoc  Committee*:    These  committees  are  convened  on 
an  as-needed  basis  to  look  at  special  issues  or  to  study 

e  problems  (e.g.,  corrective  action  situations;  emergency 
community  situations)  and  to  develop  findings*  recommenda- 
tions,  and  appropriate  alternative  methods  which  are  then 
brought;/ to  the  whole  Council. 


cnl 

'ft* 
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2.    Questions  of  Design 

At  this  stage,  your  PIC  shbuld  look  at  its  overall 
committee  structure  art*  each  subcommittee  to  see  if  .they  are: 

•  adequate  in  design; 

m     defined  most  appropriately,  given 
local  needs; 

•  -  useful  for  decision-making  purposes 

sufficiently  comprehensive  in  their 
coverage     (Do  ypu  frequently  have  to 
create  ad  hoc  committees?) 


3.    Questions  of  Membership 

The  issue  of  member  involvement  is  closely  related  to 
committee  structure.     As  a  result,  .at  this  pAht  you  should 
also  reexamine  the  membership,  of  your  PIC's  committees  to 
ensure  that  membership  is  proper.     Factors  to  take  into 
account  include: 

•  the  participation  of  as  many  members 
as  possible  on  one  or  more  sub- 

M  committees 

•  the  selection  of  at  least  "cAie  indi- 
vidual who  has  expertise  or  exper- 

-  ience  in'  the  area  Of  the  subcommittee 

.  -the  assignment  of  members  to-  sub- 
committees in  Which  there  are  motiva- 
tion and  interest  to  serve. 


\ 
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^In  summary,  the  jnost  effective  way  for  your  PIC  to  org^ize 

\ 

and  use  its  committees  will  be 'determined  by  the  character- 

✓  - 

istics  of  your  local  community  and  the  nature  of  your  Council. 


B.     LOOKING  AT  STAFFING 

/  . 

As  it  is  with  structure,  there  are  a  variety  of  possibilities 
for  staffing  PlC^ff,  ranging  from  part-time  CETA  staff  on  the 
one  hand,  to  full-time  PIC  employees  on  the  other.     Your  need  is 
to  determine  whether  the'.staf f ing  modeu  presently  being  used  by 
.your  PIC  is  satisfactory.     This  will  be  dependent  on  the  way  you 
are  redefining  your  PIC'*s  purpose,  roles,*and  functions. 

Important  considerations  here  should  be. whether  your  PIC's 
staffing  level  is  adequate, % and  is  the  division  of  labor  among 
staff  persons  equitable./  The  central  ingredient  in  £hi$  area, 
regardless  of  how  your  PIC  is  staffed,  is  to  ensure  that  your 
PlC  has  a  meaningful  process  for  overseeing  and  directing  "staff 
performance  and  a'  mechanism  or  arrangement  for  hiring,  training, 
and  replacing  staff  as  needed  to  promote  continued  competency. 


C.     LOQKING  AT  OPERATIONAL  PROCEDURES 

Another  essential  area  to  reflect  upon  is  your  PIC's 
operational  procedures  and  methods.    The  assessment  datl  will  ' 
allow  you  to  determine  whether  your  PIC's  administrative  methods 
and  processes -for  discharging  the  PIC's  business  and  responsi- 
bilities are  adequate  or  whether  they  need  to  be  altered. 
Administrative  items  that  you  should  look  at  include: 

•  Council  By-Laws     '  * 

^    Meeting  procedures  (formal  and  informal  * 
process)  ✓ 

•  Selection  and  duties  of  officers 

•  ,  Roles  and  responsibilities  o£  Council 

members,  staff. 


0 
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By-laws,  differ  depending  upon  local  circumstances,  but 


A  ' 


;£ey  usually  contain  the*  following  sedtions:    -  - 

i 

•  naiAe*  and  purpose1  of  the  PIC 

•  a  description  of  the  tK>ard's  organiza-^,    w  %  > 
ticjn,  •  powers  and  member  selection       .  *  ' 

*  process  %  * 

•  procedures  a*d  scheduling  of  members  .  * 
**  duties  and  selection  of  officers         „  4* 

cities  and  powers  of  dommittees  * 
-  •  executive  director's  responsibilities- 
^^^^      #  p3^0ceduFes  for  amendin3  the  by-laws.  .  " 

Meetings  should  be  scheduled  on  a  regular  bfcsis  depending 
v  on  iff  Council  function  and!  ned&6.     Eprmally,  PIC  meetilt^s  should 

be  run  according  to  some  forjn'of '  parliamentary  procedure. 
*  Informally^,  the  meetings  should  be  strStctured  to  promote  the 
levels  of  a£tendance. at ,  participation  during,  ^nd  interaction  ? 
within  them  which  bakes  for  an  effective  Council.  . 

An  important  part  of  Ihakmg  your  Pip  wprk  well  is  how 
you  select  officers'.  .Therefore,  you  should  reexamine,  your 
present  approach  to^make  certa*ff  that  \ it  i^  selecting  officers 
-i?I  a  way  that  is  open,   fair,  and  representative. 

In  addition,  you  should  ensure  that  the*  roles  and 
esponsibilities  of  Council  members  and  staff  are  /dear.*  In 

this  regard/  you/should  have,  at  a  mining: 

/  *  '  *  /  r 

the/by-laws  referred  to  earlier  v  ' 

1  *  *  «,  ' 

organisational  chart  fpr  the  PIC  ^ 

joH  description  for  staff  Members 

operating  procedures  for  th€#Council 

description/ of  committees1  roles  and 
responsibilities 

listing  of  gommittee  assignments' 

rmation^  on  -the  relationships  of 
PIC  to  the  prime  Sponsor,  business 
community  groups ,  etc.  *  * 
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Finally,  as'part  ofxit§  determination  with' reference  to 
operational  procedures e^yoijr  PIC  should  examine  its  present 
procepft^for  planning,  management  and  evaluation  to  ascertain* 


whether  the  prqgess  needs  revision. 


* 


*  D.     LOOKING  AT  ORGAN I ZATIQNAL  STATUS 


On^  la^t  important  concern  should  be  yotfr  PIC's  organiza- 
tional statu^.     Again,  as  with 'the  other  areas  under  this 
\  .  v 

heading,  there  is  no  single  prescribed  PIC  organizational  model, 
OpTi^ns  which  are  possible  include:  # 

•  advi^bry  position  within  CETA  Prime 
Spoi^sor's  system 

m\    incorporation  as  non-prbfit  organization 

•  segment  of  existing  local  business'  * 
organization,  e.g.,  Chamber-  of  Commerce^ 
National-  Alliance  of  Business      ♦  4  •  , 

•  multi jurisdictional*  ^C  covering^moife         *    ^     .  ^ 
than  bne  prime  Sponsorship." 

mf 

-  *  To  a  certain  extent*  your  organrzational  status  will  be 
directed  ^  yotar  purpose,  roles,  and  functions.     Factors  which 
your  PIC  should  take  into  account  A  determining  what  your 
status  should  be  include: 


I 


•  #hat  will  the  impact  b£  on  PIC  role  and 
program  dfctivities?^ 

•  What"  will  the  imp'act  be  on  o^rall.  CETA* 

^   »  prbcjj^iftming?  .  * .  ♦ 

fc^  What  will  the  impact  be  on  the  ability  of 

.  tt\e  PIC  to  represent  and  f caster  *>r ivate 

\  ^—sector  involvement? 

•  What  will^the  impact  be  on  the-  PIC  staffing 
arrangement  s  ? 

•  What  will  the  impact  be  on  the  PIC-Prime  SpcJbsor  ■ 
relationship? 
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•  How  does  the. Jjusiness  community  perceive  '  / 
•         CETA?  * ,  -v 

*  s 

-    „    •  Does  the  PIC  want  to  be  strictly  an 

^advisory  body  or  does  ,it*want  to  operate 
programs? 

Regardless  of  the  status  which  you  determine  is  correct 
for*  your  situation,  at  a  minimum,- you  should  ensure  that  you 
have  a  direct  and  clear  relationship  to  the  Prime  Sponsor, 
the  Prime  Sponsor's  advisory  council,  and  to  the  chief  elected 
official  in  your,  .community . 


CONCLUSION 

The  net^result  of  this' step  should,  be  that  your  PIC 
institutes  all  alternative  organizational  and  operational 
arrangements  nectary  to  eliminate  those  barriers  which 
^SjLhave  imDeded  effective  performance  to  date.  « 
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'  Manpower  Administration  (now  ETA),  Organization  and  Staf fiFg 
Guide  (Washington,  D.C. :  U.S.  Department  of  Labor, '  April,  1974) I 

Employment  and  Training  Administration,'  CETA,  Federal  Representa- 
tive Training  Program;  Systems  and  Management~Ana lysis  Skills  


» Block 


(Washington,  -D.C. :  National  Training  Center,  ETA,  . 
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«■  *     Formulate  Goals  and  PIC  Strategic  Plan 

The  purpose  of  this  %tfep  is  to  develop  a  ^Jrmal  written 
statement  of  your  PIC^«  gbals  and  sttategic  plan.     PIC* goals 
.are  primarily  o^  three  types:    Employment  Generating  Services; 
Participant  Services  -  Training  and  Development;  and 
Organizational.     Developing  your  strategic  plan* involves 
establishing  priority  goals,  attaching  a  time  frame  for  their 
^achievement  ,  and  making  a  rough  cletfermination  of  resources  to 
allocated  to  achieve  each  goal. 


be 


J 


fi  goal  -  broad,  unquantif ipd  statement  of  an  outcome  or  result 

which  PIC  wants  to  achieve.  > 

*    *  <■ 

•  Strategic- Plan  -  suflunary  statement  which  (1)  details  all 
the  goals  .PIC  wants  to m accomplish* in  the  upboming  year 
expressing  their  relative  importance;  and  (2)  presents 
longer-teim  goals  and  establishes  general  time  parameters 
fox  their  accomplishment. 

•  Strategic  Decision-Making  Process  ;-  method  which  PIC  chooses 
to  ;employ  to  select  its  goals  and  to  develop  its  strategic  , 
plan.  1  t     '  '  '  ^ 


fl(  j(   KnowTcdge/Capabi lines  Rating 

xReview  the  information  provided* above.     Rate  your  PIC 1 s  knowledge  and 
capabilities  with  reference  to  :thi£  step  on.  the  scales  below.  * 


•  Knowledge 


•  Capabilities 


1  

I 

1 

I 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

# 

1 

I 

 I  

I  w 

 1 

Excel-  ' 

•  Above 

Aver- 

»  Below'  - 

Poor 

lent 

Avg. 

age 

Avg. 

Based  upon  your  ratings,  decide  whether  you  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the 
next  *^tep.  .  « 
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PME  PROCESS  NOTES/ ACTtON  PLANNING  PAGE 


NOTES  (Paints  that  are  important  which  you  want  to  remember) 


QUEsTToN^(^emsonwhic.h  "you  are  unclear  pr  want  to  get  addition*! 

information^  ">\SNI. 

\  •  -      •  "  » 


7 


ACTIONS  (Specific  steps  that  you  intend  to  take  to  implement  ydur 
PIC's  PME  -process*  in  this  *airea)' 


Steps 


fte  sources/Persons 


Time  Frame 
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A. 


c 


f  .  o 

INTRODUCTION: 


^The  final  step  in  this  stage  is  converting  the  results 
of  your  assessments  and  performance  analysis,  and  your  PIC's 

'preliminary  decisions  on  initial  need  priorities,  into  goals 

•fcmd  a  .strategic  plan. 

A*  goal  is  a  broad,  unquantif ied  statement  of  an  outcome 
or  result  which  your  PIC  wants  to  achieve.    A  strategic,  pLan 
is  a-  summary  statement  which  Retails  all  of  the,.goals  you 
want  to  accomplish  in  the  upcoming  year  and" expresses  their 
relative  importance.'    In  addition/  the  plan- also  presents^ 
those, goals  which  are  longer-te^m  in  nature  and  establishes 
some  general  time  parameters,  for  their  accomplishment. 


A/  GETTING  GOALS 

There'  are  three  major  areas  in  which  your  PIG  can  set 
•goals:  , 


•  Employment  Generating  Services*, 

•  Participant  Services      Training  and 
Development 

•  Organizational         '  *  p  * 


Goal  Setting. Process  ' 

If  you've  a]  ready  established  the  priority  of  the  needs,, 

of  employers,  occupations, -and^ target  groups, a*ind  your  Pic/itself, 

and  identified  ^hy  barriers  to  be  overcome  of  reduced,  then 

establishing  your  PIC1 s .goals ' is  a  streughtforward  process 

and  can  almost  be  done  by  formula  a's  shown  below: 

f 

Goals 

*    * 

"  •  Employer  needs  +  barriers    _    Employment  Generating 
to  overcome  Services 
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* 

i 

Goals" 

W 

lW  1 

•  Priority  occupations  +  «. 

#    training  needs  +  target  # 
.  groups  *            m                        _    Participant  Services  - 

Trainina  and  beveloDment 

•  Target^articipants  +                  "•""^  TI1U  wvcauf««" 

assistance  needs  + 
barriers  to  overcome 

: 

•  PIC  internal  needs  +               _  ^aanirational 
barriers-to  pvercome-  organizational 

• 

Example  goals       they  relate  to  each  of  the  above  categories 

* 

•  * 

follow: 

AREA 

NEEDS/BARRIERS  < 

GOALS 

Employment 
Gene  rat i  ng 
Services 

.  Insufficient  labor  market 

A  -m  4-  -»      t r^-w     DTP      J  _  _  J—J  _ 

data  for  fic  decision- 
making purposes. 

•  Tc^  improve  the  quantity, 
quality,  and  decision- 
making value  of  local  labor 
market  information. 

-  Lack  of  knowledge  in  the 
busines%  community  re- 
garding PIC,  programs. 

.  To  create  and  implement  a 
comprehensive,  ongoing  . 
marketing  program  for  PIC 
activities.  » 

* 

i 

.  Inadequate  relationship 
between  public  and  pri- 
vate sector  economic 
development  efforts'. 

.  To  establish  a  coordinated, 
long-,tertFAeconomic  develop- 
ment linkage  between  the 
public  and  private  sectors 
with  the  result  of  creating 
greater  job  opportunities 
for  economical ly •disadvan- 
taged persons.  - 

• 

Participant  , 
Services  - 
Training  and 
Development^ 

.  Large  number  of  unfilled 
machinist'  positions  + 
high  percentage  of  un- 
employed black  and  His- 
panic males  between 
ages  of  18-35. 

.  To  enable  black  and  Hispanic 
males*  ages  18-35,  to 
secure  un subsidized  machinist 
jobs  in  the  private  .sector. 

• 

*  i 

J 

« 

.  Lack  of  opportunity  for 

•     local  private  sector  in- 
dustries which  employ 
significant  numbers  of 
lower  salaried  individuals 
in  entry-level  positions. 

.  To  institute  effort  with 
maior  emDlov&rs  to  uocrrade 

'  employees  from  en try- level 
jobs  to  positions  of  greater 
skill. 

f 

>  • 

".  Inadequate  understanding 
and  'development  of  general 
work  skills  and  habits 
among  youths ,  18-21. 

.  To  enhance  the  .employ ability 
of  youths,  aoes  18  to  21. 

«  * 

1 

pic 

Organization 
* 

.  Poor  planning,  management 
and  evaluation  process  - 
unclear  definition  of 

,  responsibilities  and  in- 
adequate syStem  design. 

'  w  

.  To  redesign  PIC's  PME 

Process  and  assign  definite 

PME  responsibilities  to 

both  staff  and  members. 

# 
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* 

* 

« 

• 

* 

*  ^ 
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An  Example  ' 

An  example  of  a  set  of  strategic  goals  established  by 
one  PIC  is  presented  below: 

•  To  intensify  efforts  to  directly  link* 
program  development  and  evaluation  of 
training  programs  with  the  needs  of 

•  '  business  and  industry  in  the  com- 

munity. 

•  To  increase  the  placements  of  CETA- 
/       m  ,       eligible  persons  in  the  private 

sector  through  various  employment 
and  training  programs.  r 

•  To  establish  a  coordinated^  long-term 
Oponomic  development  linkage  between 

,  :£he  public  and  private  sector  within  the 

*  community^  with  the  result  of  creating 
greater  job  opportunities  for  eco-  ** 
nominally  disadvantaged  persons* 


•  To  improve  communication  between  local 
businesses  and  educational  institu- 

,  tions. 

•  -^To  publicize  and  promote  Private  Indus- 

try Council ^activities  and  programs./ 


•  To  establish  the  relationship  between 
economic  development/manpower  goals 
and  transportation  policy: 


•  To  establish  and  str^ngtljen  linkages 
with  national  .policy-m£}£ing  organiza- 
tions that  influence  the  future  ^direc- 
tion of  manpower,  policy. 

4  •       *  -  '  * 

The  foregoing  are  pro^idepl  only  as  'examples.    <The  area9 
in  which  your  PIC  will  set  goals  and  the  nature  and  number 
of  those  goals  w-ill  be  determined  by  the  results  of  your 
own  needs  analysis  and  priorities. 

0  * 

t 

*     B .     DEVELOPING  THE  STRATEGIC  PLAN 

*         Setting  goals,  however,  isn't  the -difficult  part  of 
<>  establishing  your  PIC's  strategic  plan.    The  more  demanding 

O  ■  I  * 
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task  is  deciding  what  your  PIC's  mix  of  goals  should  be  and 
which  goals  should  take  precedence.    What  should  be  accomplished 
this  year  and  what  should  be  postponed?    How  should  this  year's 
resources  be  apportioned? 

This  determination  is  truly  your  option.     The  CETA  Regu- 
lations only  stipulate  the  percentages  of  your  Title  VII 
funcfs  which  can  or  #have  to  be  allocated  to  each  category  of  4 

activities: 

*  * 

•  Direct  participant  services-  at  least  50% 
,  •    Employment  generating  services  -  up  to  30% 

•  Administrative  activities  -  no  more  than  20% 

Economic  development  activities  -  10%  incen- 
tive for  involvement. 

Within  those  guidelines^  it^s  up  to  you  to  determine  what 
relative  priorities,  mix  ^f  goals,  and  eventual  distribution 
of  dollars  can  best  meet  the  employggijfe  and  straining  needs  » 
within  your,  community.     Factor  §  that  jshould  be  tatken  into 
account  in  making  these^ decisions  between  and  among  goals  are:  * 

•  appropriateness  to  your  PIC's  mission 

•  appropriateness  to  economic  conditions 

•  immediacy  6f  need  # 

« 

•  responsiveness  to  employer  needs 

•  responsiveness  to  target  group  needs 

•  acceptability  to  community 

•  obstacles  to  achievement  r 

i  •    tradeoffs  iri  outcomes  associated  with 
selection  of. one  -goal  over  andther. 


4 


strategic'  Decision-Making  Process  *  '  ■ 

As  with  the  earlier  steps  in  this  stage,  the  process 
you  employ  to  make,, your  decisions  on  the  strategic  plan  should 
be  the  one  which  will' work  best  for  you  given  your* Council ' s 
and  local  circumstances.     A  possible  method — similar  to  that 
for  defining  your  PIC's  mission  and  functions — for  developing 

•  ■  •  ) 

•  25, 


the  strategic  plan  is/ described  below:.  r 

a.  Have  staff  develop  brief ing' papers 
describing  the  results  of  the:  ' 

-  assessment  of  PIC  program 
p  e  r  f  o  r  mahc  e  * 

A\   •  -  assessment  of  organizational 

and  operational  performance 

*  * 

-  initial  need  priorities 

•  EGS 

.  target  groups  , 
1  .  occupations 

.  participant  service  needs 

-  PIC  mission  and  functions 

and  listing  as  many  goal  options  as 
possible  based  upon  these  results. 

b.  Disseminate  papers  to  all  Council 
members.  *  • 

c.  Have  Council  members  read  and  ask 
them  to  nhpnk  off  t-.heir  priority *goals 


/ 


for  the  PIC  from  -a  list  of  goal  options. 

d.  Convene  the  meeting  and  have  all  Council 
members  report_out  goals  selected. 
Record  them  on  a  .flipchart.     Use  hash 
marks  to  indicate  goals  selected  by 
more  than  one  member .  » 

e.  After  all  members  have  reported 

have  group  discussion  focus  primarily 
on  those  goals  on  which  &  number  of  the 
members  agreed, 

f .  Following*  the  discussion,  attempt  to  achieve 
consensus  on  those  goals  which  should 

be  addressed  in  the  upcoming  year  and  which 

Should  be  longer  term. 

*■  .  •> 

g.  After  this  determination,  attempt  to 

get  a  priority *ranking  of  goals  for  the      ,  * 
upcoming  year  through  consensus. .   If  this 
is  impossible,  ask  members  to  vote  to 
determine  rank  order  Jror  all  goals  for 
year.  * " 

'v 

s       *       .  /         t>  t 

Once  a  priority  order  is  established, 
end  the  meeting.  v  , 
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i.     Prepare  results  of  the  meeting — both  upcoming 
year  goals  in  order  and  other  long-term  .  ^ 
goals  for  PIC— and  sertd  to  the  members  along  • 
with  tot^l  budget  estimate  information. 

j.     In  advance  of  the  meeting,  ask  each  member 

to  allocate  a  percentage  of  the  budget  to  each 
goal*. 

k.  ^Convene  a  second  meeting  to  allow  members 
to  allocate  ro^h  percentages  of  the 
budgcet  to  each  goal*. 

1.     Have  members  report  allocati6ns. 

Recor3  them  by  each  goal  on  a  flipchart.  « 

m.    Have  .members  discuss  their  relative  « 
allocations  to  attempt  ^to  acliieve  a  con- 
sensus ori  the  -budget  amount  to  be 
allocated  for  each  goal  without  averaging. 

•  ** 

n.     If  this  is  impossible,  average  all 

responses  to  eabh  goal.     Have  members 
review  averages.     Then,  have  members 
«   discuss  again  to  seek  tradeoff  pbsi- 
tions. 


o..    Obtain  either  an  absolute  percentage  or 
percentage  range  allocable  to  each  goals 

p.     After*  the  meeting*  prepare  a  written  state- 
ment describing  PIC's  strategic  plan. 

%?  '     .  - 

Regardless  of^  the  method  that  your  PIC  employs'  to  establish 
its  strategic  plan,  once  your  PIC  has  reabhed  consensus  on  its 
goals,  their , relative  priorities,  and  the  rough  percentage  of 
the  budget  Ub  be  allocated^ to  each  goal,  this  information 
should  be  recorded  as  the  PICfs  strategic  plan  for  the  year. 

In  addition,  the  plan  should  list  all  other  longer-term 
goals  which  are  Important  for  your  PIC  and  which  v^ill  be 
addressed  in  succeeding  years.     The  portion  of  the  strategic 
plan  relating  to  next  year  provides  the*  initial  substantive  ' 
parameters  l:or  the  full  scope  of  programmatic  activities  to  be 
undertaken  by  your  PIC. 
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Manpower  Administration  (now  Employment  . 
and  Training  Administration  [ETA] ) ,  Manpower 
Program  Planning  Guide  {Washington,  D.C.: 
U.S.  Department  of  Labor,  April,  1974) . 
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C.     ESTABLISH  PROGRAM  SCOPE 


— v- 

Purpd se : 


To  translate 'the  broad  goals  and  directions  • 
set  out  in  the  strategic  plan  into  specific 
program  objectives  and, pro jects  for  achieving 
them. 


to 


(  ' 

Define  Appropriate  tactics 
Draft, PIC  Program  Objectives 
Specify  Projects  and  Scope  - 
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Define  Appropriate  Tactics 

The  purpose  of  ' this  step  is  >to, decide  what  tactics  you  wij.1 
employ  to  achieve  the  programmatic  goals  which  you  have 
set  in  ycfur  PIC's  strategic  plan.    Programmatic  tactics  can 
.be  separated  into  two_majbr  categories:    participant  services 
and  employment  generating  services. 

(  '  -  . 


Tactic  -  general  means  or  broadly -formed  approach  fof' 
achieving  a  goal  or  an  end. 
Participant-Related  Tactics  -  classroom  training  -  vocational? 
classroom ^railrtirg--  empioyabilityr ^n-the- job  tYainmg? 
upgrading?  retraining?  employment  and  training  services? 
and  supportive  ^services  (e-g.,  health  arvi  child  care) ^ 
Employment  generating  Tactics  -  labor "market  analysis?  mar- 
keting? economic  and  comBRmtty  develay^en£;  ineenfetve  programs- 
(e.g.,  TJTC)?  and  linkage  creation-  *  -  , 


ff(  jj   Knowledqe/CapiblUties  Rating 
Review  the  infornatioiLprovided  above/  Rate  your  PIC's  knowledge  and 
capabilities  with  re^nce  to  this^  step  ..on  the  scales  below. 


*  * 


•  Knowledge 

•  Capabilities 


_1_ 


_1_ 


Ejcce 
lent 


Aboue 
Avg. 


Aver- 
age 


Below 
Avg. 

1 


Poor 


Excel- 
lent 


Above1 
Avg. 


Aver- 
age 


Below 
Avg. 


Poor- 


'eased  upon  your  ratings,  decide, whether  you  need  to  read  the  content 
!nrorma??o"nyon  this  s?ep  .which  follows  immediately  or  proceed  to  the 
next  step. 
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QUESTIONS 

litems  on  which  you  *ar«e  unclear  < 
information) 

•  ■ 

0  * 

ta*»  w3Mt  ,to 

vV 

1  £ 

*         *  • 

• 

get  additional, 

• 

t 

ACTIONS   (Specific  steps*  that  you  intend  to  take  to  implement  your  # 
PIC's  PME  process  in  this  area) 

Resour«c£s/Persons 


t.       Steps  ' 


Time  Frame 


r  ' 


INTRODUCTION 


The  first* step  toward  converting  \our  PICVs  strategic  ^ 


plan  into  a  wofkable  reality  is  defining  what  tactic  (s) 
is^are)  most  appropriate  and  should  be  used  to  accomplish 
each  goal  that  you  have  set.  A  tactic  ^is  a  general  means 
or  broadly- formed  approach  for  achieving  an  end. 


rt> 


A.     CATEGORIES  OF  TACTICS 


The  range  of  tactical  options  available  to  your  PIC 
is  Wide.    Again,  as  with  your  goals ,  these  options  can  be 
broken  down  into  the  categories-  of  participant'  services  and 
aent  generating  services. 


Participant-Related  Tactics 

Major  participant-related  .tactics  include: 

'V- 1  ■ 

•  .         •  Classroom  training  (vocational) r  training 
:  conducted  away  from  a  job  that  provides  the 

*  .        participant  with  specific  technical  knowledge 
and^ skills  .necessary  to  perform  successfully 
in,  a  }ob*.  ■  ' 

•  Classroom  training   (empfoy ability) :  training 
Conducted  away  from  a  job  that  providea^h^ 

•  ^  . participant  with  the  ftind&mental  knowledge, 

f<>*  ~  .  skills,  abilities,  and  attitudes  necessary 
f  ,to  secure  ancf  hold  an  unsubsidized  private 

seqfrpr  job.  * 

.  •  On-the-Job  training ^  training  aftd" soger-, 
«♦  .  vision  provided  To  participants  hired  *  by 

0  private  sector  firms  to  enable  them  to 

dfevelop  air  necessary  occupational  skills 
while  employed  in  a  productive  job  in  the 
private' sector .  j 

•  Upgrading;     skill  training-  (either  "(JlassrocJm  . 
%  /              oi  OJT)  of  individuals  locked  in  low  paying, 

dead-end  jobs  -to  enable  them  to  qualify  for 
positions  requiring  greater  akill  within  the 
-  saftae  business  enterprii 
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•  Retraining:     skill  training  (either  classroom 
t>r  OJT)  in  occupational  skilk  shortage  areas  1 
to"  teach  participants  new  skills  distinct  from 
those  already*  possessed •  *t 

•  Employment  and  training  services;     services  to 
^enhance  an  -individual's  capabilities  and  oppor- 
tunities for  securing  urtsubsidized  employment. 
These  services  include  but  ar.e  not  limited  to: 

-  orientation  to  world  of  work 

,  -  counseling  (employment .related  and 

testing)  ^ 
.  -  employabiJLity  assessment 

-  Job  development.  4 

-  job  search  assistance  - 

-  job  referral  and  placement.  . 

-  'Target  e8  Jobs  Tax  Credit   -  eligibility  • 

*  determination  / 
-"Vocational  exploration  programs 

•  Supportive  services: '  services  to  eliminate 
possible  barriers  to  successful  employment. 
Services  include  buf  are  not  limited  to: 


-  health  care  and  medical  services  i 

-  child  care  \ 

-  transportation  / 

-  temporary  shelter 

-  financial,  counseling  and  assistance 

4, 

, "     Within  this  general'  range  of  participant  tactics,  there  " 
are  two  specialized  tactics  which  deserve  highlighting  because  4 
of  tjia^r^particular  relevancfe"  and  currency.     These  tactics  are: 
.customized  training,  and  pre-employment  and  ^b  search  training. 

1.     Customized  Training  m 

Customized  training  can  be^ither  classroom  training  or  OJT 
or  a  combination  of  the*  two.     It  differs  somewhat  from  bdth, 
however,  not  because  of  tfie  delivery  method,  but  because  of  the 
method  of  development.  m 

Customized  training  program^  are  normally  developed,  by 
having  a  local  employer  or  employers  identify  "high  demand" 
occupations  in  whiih  akill  training  is  required,  and  then 
participate  in  the  curriculum  development'  process  fco  ensure  a 
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training  prqgram  which  is  relevant  to  ,the  local  business 
needs.     The  customized  training  program  oftentimes  involves 
the  participation  of  the  same  employer (s)  in  providing 
instructors  and/or  training  facilities  at  company  sites 
and  OJT  slots  and  eventually  full-time  unsijbsidized  jobs 
for  program  participants. 

The  major  advantage  for  your  PIC  of  employing  the  cus- 
#  tomized  program  tactic  is  tha't  it  ensures  a  program  which 
is  cUrectly  responsive,  to  the  needs  of  the  private  sector. 
To  many,  :  this  tact^jfe^  epitomizes^thj^ intention  of  the  fsiV. 


*  2.     Pre-Employment  and  Job  Search  Training 

Pre-employment  and  job  search  tactics  involve  no  occu- 
pational  or* skills  training.     Instead,  they  are  designed  to 
make  the  participant  employable.     That  is,    (1)  to  prepare 
the  participants  for  work  by  developing  their  motivational, 
self -assessment ,  resume  writing,  and  interviewing  skills; 
and  *(Z)  to  enable  the  participants  to  secure  unsubsidized* 
privat^Ssecfc^  jobs  on  their  own.    The  major  advantage  to 
your  PI£  of  employing  the  pre-employment  and  job  search  . 
taotic^ is  that  it  is  relatively  low  cost  and  quick  turn- 
around in  nature.  ^ 

The  limitation  of  the  tactic,  of  course,  is  that  it  is 
|Only  appropriate  with  those  participants  who  are  already 
job  ready  in  terms* of  skills  and  abilities,  and  for  those 
occupations  which  do  not  have  high  skill  requirements. 


Employment  Generating  Tactics 

The  tactics  available  to  your  PIC  under  the  Employment 

Generating  'heading  are  even  more  varied  than  those  which^are 
« 

participant^related*    Som6  of  the  most  common  major  tactics 
your  PIC  can  choosy- from  include:  ,  < 

r^abor  -Market  Analysis:     studies  of  local 
A  businertes  to  identifylthe  particular  labor 

*     marketjfieeds  withft*  the  community. 
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•  Marketing;  activities  to  expand  awareness, 
explain  benefits,  and  provide  assistance  to 
local  businesses  to  increase  'their  partici- 

%        pation  in  *CETA  programs. 

•  Economic  and  Community  Development;,  efforts 
to  attract,  retain,  and/or  expand' businesses 
to  enhance  economic  and  cojjmmnity  conditions 
within  local  arete.  w 

•  Incentive ' Programs ;     programs  created  to 
offer  or  to  promote  special  benefits,  such 

as  Targeted  Jobs  Tax  Credit,  to  induce  private 
sector  employers  to  hire  CETA-eligible  partici-- 
.  pants. 

•  Linkage  Creation:     efforts  to  forge  new  rela- 
tions  with  existing  institutions  (i.e., 
government  agencies,  business  groups,  cote-  * 

•  munity  groups,  economic  development  grasps, 
labor  unions,  local  planning  bodies,  and  ^ 
.schools/colleges)  to  improve  employmentjarid 
training  capabilfties  within  the  locai^^^rnmunity . 


B.     SELECTING  YOUR  TACTICS 

The  foregoing  list*  of  tactics  for  participant  and 
employment  generating  services  are  meant  to,  be  instructive 
and  not  exhaustive.     These  tactics  are  not  important  as  ends 
in ^themselves,  but  only  as  they  relate  to  the  goals  you  have 
set  for  your  PIC.     To  a  large  degree,  each  of  your  goals  will 
imply,  and  the  analysis  whichj>£pceded  its  development  vplil 
enabl^  you  to  select,  a  primary ' tactic  which  will  be  appro- 
priate to  your  end.  *  , 

V 

Examples 

The  examples  which  follow  show  the  relation^.betveen  goal 
and  barriers,  and_  taptics  which  might  be  ^selected  in  response 
to  them.  -  -  * 
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t  Tof  enable  unemployed 
black  or  Hispanic 
males ,  ages  18-35, 
to  secure  imsubsidized 
clerical  jobs  in  the 
private  sector. 

To  improve  the  quantity, 
quality,  and  decision- 
making value  of  local 
labor  market  informa- 
-  tion.  c 

To  establish  a  coor- 
dinated long-term 
economic  development 
linkage  between  the 
public  sector^and 
private  sector. 


PRIMARY  BARRIER 

Knowledge  and  skill 
deficiency. 


Inadequacy  of  stan- 
dard source  data. 


Inadequate  present 
relationship 


.APPROPRIATE  TACTIC (S) 

Classroom  training:  f 
vocational  - 
machinist  skills 


Labor  market  analysis  - 
1  local  employer  sufvey 


Linkages  and  economic 
development 


ft 

>0 


Process 

^^^pWen  this  direct  relationship  between  goals,  barriers, 
and  tactics,  the  process  which  yofir  1*1  C  can  employ  td  select 
tactics  is  fairly  straightforward.     You  can,  use  the  general 
format  set  out  below  to  select  your  tactics.  *  a 


GOAL 

BARRIER 

TACTIC 

■7 — ■  r- 

0 

Your  PIC  should  select  -a  primary  tactic  to  acnieve  each 
goal  that  you  have  set.    The  goal  plus  the  tactic  provide  the 
basis  for  setting  your  PlC's  programmatic  objective  and 
beginning  the  process  of  shaping  your'  PlC's  total  program  of 
activities  and  services. 
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National  Alliance'of  Business,  Roles, 
Organization  and  Programs:  Basics  for  the 
Private  Industry  Council   (Washington,  D.C. 
National  Alliance  of  Business,  1981). 

Chamber  of  Commerce  of  the  United ^States, 
The  New  Private  Sector  Initiative  Program 
(Washington,  D.C.     ^     :  "~ 
the  U.S.,  1979) . 


Chamber  of  Commerce  of 


Mayor  s  Office  of  Manpower,  City  of  Chicago,  Planning  and 
Evaluation  under  CETA  (Washington  D.C:  U.S.  Department  of 
Labor,  January  1976). 
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Draft  PIC 'Program  Objectives. 


The  purpose  of  this  step  is  to  establish  ydUr  PIC's  program 
objectives  for  the  fiscal  year.    Objectives* are  specific 
statements  of  .what  your  PIQ  wants  to  accomplish.  There 
are  two  major  types  of  objectives:  Participant-Related; 
Employment  Generating  Services  Related.  ^ 


a 


•  Objective  -  a  quantifiable,  or  in  some  other  way  measurable.* 
'  expression  of  what  is  to' occur  specifically  as  a  result  of  ' 

•  'a  program  tactic.    '         *  - 

•  Participant-Related  Objective  -  answers  the  questions  of 
who,  how  many,  what  means,  and  primary  results  expected.  v 

•  EGS-Related  Objective  -  answers  the  question  of  "how  well", 
in  terms  of  time,  quantitative  measure,  qualitative  value, 
and  cost  efficient/effectiveness/effort  value.  ; 
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Review  the  information  provided  above.  Rate  your  PIC's  knowledge  "and 
capabilities  with  refe^nce  to  this  step  6n  the  scales  be'low.  * 

L  1_   I    i  t 


wledge 


Excel- 
lent 


•  Capabilities 


Above 
Avg. 

I. 


Exce 
le 


Above 
Avg.* 


Aver- 
age 


Aver- 
age 


Below 
Avg. 


Poor 


Below 
Avg. 


Poor 


Based  upon  your,  ratings,  decide  whether  you  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the 
next  step.  . 
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PME  PROCESS  NOTES/ACTION  PLANNING  PAGE 


NOTES   (Pointy  tha 


t  are  important  which  you  want  to  remember) 


"QUESTIONS  (Items  on  which 'you  are  unc 
  ■  information) 


T^r^r^San^Toget  additional 


VTIOHS  (Specie  steps  That  V-  -tend  to  taKe  to  ISpliiiHt  your 
—    PIC's  PME  process  in  this  area) 


Steps 


Re  sources/Persons 


Time  Frame 
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INTRODUCTION  _ 
.    *  ^* 

*  / 

Once  you  tiavt*  goals ^and  tactics  related  to  them,  you 

translate  this  information  into  your  PIC's  program  objec- 
tives for  the  fiscal  year.     Program  objectives  t§ll  you  what  . 
your  PIC  is  going  to  do  and  how  well .  > 

program  objective  should  be  either  a  quantifiable, 
or  in  some  other  way  measurable,  expression  of  what  is  to 
occur  specifically  as  a  result  of  a  program  tactic.  Your 
PIC's  program  objectives  help  to  narrow  the  focus  of  your 
goals  and  provide  the  bridge  between  the  more  abstract  or  f 
conceptual  strategic  planning  stage  and  the  more  ^fjncrete 

sfage  of  operational  planning.     They  also  provide  criteria 
« 

against  which  to  measure  programmatic  performance  at  the  • 
end  of  the  Fiscal  Year. 


A.     SETTING  PROGRAM  OBJECTIVES  ' 

It's  useful  to  look  at^program  objectives  in  the  two 
categories  described  earlier:- 

•  Participant-Related  , 

•  Employment  Generating  Services  (EGS)  ^ 
/  Related^       \  m 

jf      The  process  of  setting  objectives  for  each  category  is  similar, 

but  the  actual  mechanics  are  slightly  different.' 

Participant-Related  Objectives 

>  Participant-related  objectives  should  answer  at  least 
four  basic  questions:,  '  ^ 

.  Who  -  description  of  people  to  be  served 

How  Many  -.number  of  people  to  be  served 

What  lUtms  -  tactic  (primary  service  or 
activity  to  be  provided) 

Results  -  what  primary  £inal  outcome  is- 
expected. 
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An  example  of  a  participant-elated  objective  based 
upon  a  goal  set  earlier  follows: 

Goal:  *  To  enable  unemployed  black  or  Hispanic  , 
males ,*   ages  18-35,    to  -secure  unsub- 
sidized  machinist    jobs  in  the  private 
sector.  • 

The  goal  tells  you:  ... 

Who:     black  and  Hispanic  males,  ages/18-35  1 

Results :    unsubsidized machinist  jobs  in  the 
private  sectpr.  " 

.  The  tactic  tells  you:  .  * 

* 

Wtyit  Means :    classroom  training. 

At  'this  point,  it's  up  to  your  PIC  to  answer  the  question 
of  how  many  in  <5TGer  to  create  a  program  objective.  Let'sL 
say  that  your  Council  answered  that  question  as  follows: 
^  How  Many:     75.  " 

Then,  your  PIC's  program  objective  related  to  this  goal 
would  be:  " 

m    Objective:     To  enable,  through  classroom 
training,  75  unemployed  black  and  Hi'spanic 
males,    ages  18-35  ,    to  secure  unsubsidized 
machinist jobs  #in  the  private  sector. 

*  . 
While  the  above  is  sufficient  for  a  program  objective, 

if  it  chooses,  your  PIC  ca/i  make  its  objective  even  more 

precise  by  answering  other  questions  regarding  results,  or 

how  well,  e.g.,  at  what  cost,  quality  of  job.     An  expanded 

objective  incorporating  answers  to  these  -questions  follows: 

To  enable,  through  classroom  training,  \ 
75  unemployed  black  and  Hispanic  males, 
ages  18-35,  to  secure  unsubsidized  private 
sector  positions  having. an  average  wage  of 
$5.25/hr.  at  an  average  cost  of  $1, 500. 00 
per  participant.  , 

%  "  < 

^tour  PIC  should  make  its  participant-related  progr'am.  objectives, 
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as  precise  as  you  feel  they  need  to— be^for-  program  manage- 
ment purposes*     The  advantage  of  more  detailed  program  oj^ec- 
tives  is  that  they  provide  clearer  direction  .for  all 
planning,  management,  and  evaluation  activities  to  follow. 

_     \       '  : 
EGS-Related  ObjectivesL  '  / 

Given  the  broader  scope  and  somewhat  less  tangible 
nature  (e.g.,  linkages,  improved  business  relations)  of 
Employment  Generating  Services,  as  opposed  to  those  activi- 
ties which  are  directly  participant-related,  it  will  probably 
be  more  difficult  for  your  PIC  to"  set  objectives  from  goals, 
related  to  them.     It's  just  as  necessary  and  important,  however 

fortunately,  setting  your  EGS  "objectives  can  be  as 
straightforward  a -.process  as  sfetting\your  pSrticipant- 
related  objectives.     Your  goal  tells  you  the  general  end  - 
you  want  to  achieve  and  your  tactic  tells  you  what  you 
need  to  do  in  order  to  achieve  that  end  (e.g.,  conduct  a 
local' employer  survey).*    To  develop  an  EGS-related^sJijec- 
tive,  you  need  only  to  determine  "how  well."    To  determine 
"how  well"  you  should  answer  the  following  questions: 

•  By  what  time  or  within  what  tiitle  frame 
should  the  project  be  completed?  * 

N  •  Is  there  a  quantitative  measure  or  value 
which  can  be  assigned  to  the  project* 
(e.g.,  contacts  to  be  made',  reports  to 
be  produced,  meetings  to  be  held)? 

*  • 

•  Is  there  a  qualitative  value  which  can 
be  assigned  to  the-  project  (e.g.,  number 
of  acceptable  errors,  adherence  to  some 
established  external  criterion,  accept- 
ability to  the  Council)? 

•  Is4there  a  cost-efficient/effectiveness/  . 

•  ef fogrt  value- which  can  be  assigned  to 
the  project  (e.g.,  cost  per  report 

,  produced,  cost  per  person  placed,  num- 
ber of  person  days  to  be  spent  on  projeqt)? 

Three  examples  of 1 EGS-related  objectives  pertaining  to  % 
goals  described  earlier  and  answering  all  of  jthe  above 
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questions  follows 


Goal :     To  improve  the  quantity,  quality, 
and  decisiQn-making  value  of  locil 
labor  market  information. 

Objective;     Tp  complete,  within  6  months, 
a  loc§l  empfoyer  survey  of  100  selected 
leading  industries  in  order  to  identify 
valid  and  reliable  data  on  entry-level 
job  opportunities  and  salary  level  of 
positions  at  a  total  survey  cost  of  no 
more  than  $10,000.00. 


'  Goal ;    .To  establish  a  coordinated  long- 
term  economic  development  linkage 
between  the  public  and  the  private 
sector  with  the  long-term'  result 
of  creating  job  opportunities  for 
economically  disadvantaged  persons,  - 

Objectives :    .To  create  a  joint  committee 
to  'be  composed  of  representatives  from  public 
sector  economic  "development  agencies  and  pri- 
vate sector  investment  firms  which  will  ^ 
^develop  within  the  next  9  months  comprehensive 
local  economic  development  strategy  to  benefit 
CETA-eligible  participants  at  a  cost  of  2 

person  months  of  effort  from  PIC  staff* 

gTo  coordinate  with  the  City  Development 
'Department  to  ensure  that  20%  of  construe-  - 

tion  jobs  being  generated  on  the  two  downtown 
.  redevelopment  project^^^  supported.- primarily 

through  EDA  and  UDAC  ^rnds,  are  set  aside  for 

CETA-eligible  participants. 

Again,  your  EGS  objectives  need  not  be  this  detailed. 
At,  a  minimum,  however,  for  each  EG£v-r elated  objective  your 
PIC  should  specify  the  end  product- or  result  it  expects  and 
associate  at  least  one  "how  well/1  measure  (i.e.,  time,  quan 
tity,  quality,  ,cost)  with  it. 
m 
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B.     DEVELOPING  YOUR  OBJECTIVES 

To  set  your  PIC's  objectives,  all  that  you  have  to  do 
is  apply  the  procedures  described  in  tl\e  preceding  pages  to 
each  goal.     A  format  fox*  doing  that  is  presented  below. 

¥  • 

Participant  Goal :  x  

.     Who:  m  _J_  =  ■ 

How  Many:     ~~  / 


What  Means: 


Primary  Result: 


How  Well  (e.g.,  avg.  cost/participant,  avg.  wage) 


EGS  Goal: 
Time : 


Quantity : s  *  


Quality: 
Cost: 


•  The^e  are  no  absolute  numbers  of  program  objectives 
that  ought' to  be  set  by  any  PIC.     The  essential  considera- 
tion should  be;  as  noted  earlier,  that  your  PIC  set  at  least 
one  objective  for  each  gQal/tactic  that  you  intend  to 
undertake, 
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Manpower ' Administration  (now  Employment  -and 
Training  Administration  [ETA] ) ,  Manpower 
Program  Planning  Gu£de  (Washington,  D.C. :  - 
U.S.  Department  of  Labor,  Apr^j.^  1974). 

,  •  > 

♦Mayors  Office  of  Manpower,  City* of  Chicago, 
Planning  and  Evaluation  under  CETA  (Wash*-  - 
ingtpn,  D.C:  IKS.  Department  of  Labor, 
Januaty  ^6)  , 
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Specify  Prj^ects  and  "Sc6j^ 


Ttys  purpose  of  this  step  is  to  identify  thet  specific 
projects  which  your/ PIC  will  support  in  this  fiscal  \ 
year..  This  involves:     determining  the  number  and  type'of 
projects  to  be  ryn;  defining  the objectives  and  scope  of 
each  project;  and  allocating  resmirces  to  Jh,e  projects.  * 


Projects  -  discrete  services  and  activities  to  be  sponsored 
through  PIC  in  order  to  achieve. its  programmatic  objectives'. 

Project  Scope  -  specific  objectives  to  be  accomplished  and 
services  or  activities  which  Vre  £o  comprise  each  -p'roj'ect. 

•  Project  Resource  Allocation  -„ estimatiort  of  amount  to  be 
expended  on  each  pro ject* to  endure  cost  feasibility  and  • 
compliance  witl> or iginal  program  plan,  ' 


> 


Mi  U   Know! edge/Upabin ties  Ratify  t^>- 

Revifew  the  information  .provided  above.  Rate  your  Pie's  knowledge  and 
capabilities  with  reference  to  this  step  on  the  Scales  below ; 

^  }  "I 


•  Knowledge 
•.  Capabilities m 


.  .1  

I 

J  . 

* 

Excel- 

Above 

Aver- 

Below , 

Poor 

lent 

Avg  • 

age 

Avg. 

0 

1 

• 

J 

 1 

Excel- 

Above % 

Aver- 

Below 

lent 

*  Avg. 

age 

Avg. 

—    .  * 

"'.Based  -upon  your  ratings,  decide  whether  you  need  to  read  the  «flntent 
^information  on/ this  step  which  follows  immed-iafre^  or  proceed  *>  the 
ne^ti  step.  •  w 
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PME  PROCESS  UOTES/ACT I ON. PLANNING  PAGE 


NOTES™7PointTThat^?ein^r  t  art  t  .which  you  want  to  remember) 


X 


} 


■■^■^Q^^Item^^  wa'nt 'to  get  additional 

,  informati^i)  fc. 


\ 


V 


1 


/ 


i 


Time  Frame' 


AcTToN^ffiecm^^teTs^ha^yoT^ntend  to  take  to  implement  your 
.  *PIC's  PME  process  in  ,thi§  area) 
Steps      '  '  Resources/Persons 

— .  1- 


^1 


INTRODUCTION 


The  last  ste^ in  the  development  of  the  preliminary 
•general  design  of  your  PIC's  program  for  the  fispal  year  is 
to  identify  those  specific  projects  to  bfe  supported. 
'  Projects  are  the  discrete  services  and  activities  to  be  . 
sponsored  through  the  PIC  in  order  to  achieve  its*  program- 
njatic  objectives*     There  are  three  tasks  ^involved  in  project 
specification:      "  *  * 

•  ■  Determine  number  and  types,  Dfects. 

•  Define  objectives-  and  scope  of  each  project". 

•  Allocate  resources  to  projects. 


r 


'A.     DETERMINING  PROJECT  NUMBER  AND  TYPES 

Pro jects  are,  derived  directly  from  programmatic^bjec- 
ti-Ves-  v  In  some  instances ,  you  may  have  only  afre  project 
pet  objective.     In 'others,  you  may  have  as* many  as  three 
or*%our  projects.     This  fact#ds  represented  in  thre  .f low  v 
chahrt  on  the  'following  page  which  suwmarizes  PIC  activity 
to  this  point  in  decision-making  terms* 
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PIC  Planning  -Decision-making  Flow  Chart- 


t 


\ 


^Pf  babj 


efcamplrf  of  a  program  objective  for  which  you  would 

have  Lnly  bne  project  is  the  EGS .objective  related  'to 

et  analysis^  ^  *  ' 

•  •  ' 

•Objective:    To  complete,  within  6  months, 

alioca}  employer  survey  of  100/ selected  .  • 

liading  industries  in*  order  to  identify 

vauid  and  reliable  data  on  entry-16vel 

joDsoppor tun i ties  and. salary  level  / 

positions  at  ar  total- survey  cost  of  no 

'than  $10,GO0.0<?7\ 
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The  so^e  project  to*  be  undertaken  with  reference  to  this  objective 
would  be  a  local  employer  survey.  •  , 

On  "the  .other  h&nd,  there  'are'  ofcher  areas  in  which  your 
PIC  might  iijjtiate  multiple  projects.'   For  example,  referring 
bafck  to  the  participant-related  objective  for  machinist  posi- 
tions: V  ' 

To  enable,  thrbugh  classroom  training,  * 
*         '-75  unemployed 'black  apd  Hispanic 

Njnales,     ages  18-35  ,    to  secure  Hinsub-  K 
s^dized  private  secitor  positions  having 
an- average  wage  6\  $5n25/hr^,  at  an 
average  cost  off $1 , 500 . 00  per  partici- 
pant. 

It  is  possible  that  your  VlC  might  want  to; split  up  this 
objective  into  two  projects  or  more  with  each  project  being 
-operated  by  a  different  service  deliverer  (e.g., ^community 
college;   private  sector  ..proprietary  institution,  non-profit 
community-teased  organisation)  . 

As  another  example  of  a  program  objective  which  would 
be  met  through  multiple  projects,  let's  look  at  an  objec- 
tive  related  to  ari  example  goctl  stated  earlier  ir^the  text: 

teoal :     To  institute  a  program  .with,  ma  jor  v 
local  employers  to  upgrade  employees  from  * 
entry-level  jobs  to  positions  of  greater 
skill  and  higher  salaries.   •  ( 

Objective;    To  institute  a  program  with 
i     major  local  employers  to  upgrade  at 
least -20d  employees  from  entry-level 
jobs  to  positions  of  greater  skillis 
;        and  higher  salaries.-        ^  L  , 


•         Obviously ,  .  this  objective  calls  foj:  the  designation  of- 
a'. specific  number  of  .emplo^se^  by  name  and  participants  per 
employer  for  'the  upgrading ^program  as  an  integral  part  of 
the  plianning  process.  ~~  Y 


B.     SETTING ^SCOE£\AND  OBJECTIVES 


w    °     Once  the  nunjber  and  type  of  projects- for  each  objective     1.  ^ 
are  clarified,  .your  next  task  is  to  establish  the  specif it 
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*4 


•objectives  ^nd  scope  for  e^ch  project.     Tl\is  process  begins 

t      '    s  "* 

r    wi£h  breaking  the  prograp  objectives  down  and  assigning  specific 

object iv#S  to  each  project.     Thip  is  done  by  answering  thS same 

-questions  i^osed  earlier .to  establish  your  -PIC  1 s  program  objeo  

ti^es  related  to  I^rticipaift  and  Employment  Generating  Services, 


X. 


Participant  Employment  Generating.  Services 

Who^.       t  Primary  Result? 

How. Many?  .  How  Welt?. 


What/Mean^?  -  Time? 

"irimaft.  Result?  •  *  *  '  "'J1*?^*?*/ 
f      .      ■  x  Quality?  r 

Hpw  Well?,  \  .4-  Cost? 


,       -  ;  Th6  end  result^of  this  process  shduld        a  clear  state- 
-^'me'rft.  for  e^h*pro}ec-^|^f  the:  -  „ 

••~Tv  £  &     principal  outcome  in  jneasurable  terms  r 
>    ;  •   ^•r    primary  service, or  assistance  to  be  delivered.  » 

v^  V     1  •   %  f  •  •      \  * 

Est^blik^^g  ^f object  Sflope  -  J 

*£s     'The'  sewnii  ^a^lc^re  ^s-to  determine  the  basic  Scope  or 

§erV\ces  ajfi I ?lt  1  e s  to.  comprise  eaeiiv  pro ject .  ,  This  is 

\  relatively  simpie  for  your  PIC's  EG£f  projects,  since  they  will' 

, o^dinarilvco^sist  *6f  a  single  primSrJ  activity  (e.g.,  labor 

'  market ' analysis,  /linkage  project,  economic  development  effort) 

'.and  a  set  of  sub-$cl;wities  directly  linked  to  that  project. 

,  However,  your  participant-serving  jproject's  will'probably  be 

Somewha^  more  -complex,  since  they  .must;;  bfe.  designed  to  meet  the 

specific  needs  ojf  youfptefirget  gtfdups.  4ti 

As;  noted  -  earlier  in  Stage  B ,  Step  d,*,t&rget  group  participant 

^y&ee&s^tan  ,be  either  of  a  training" and/bf  other  service  nature. 

Therefore,  ^ven'if  3.  primary  project  is* 'classroom  training  or  * 

OJT,   at  this  point  ydu  need  to  decide  wrflt;other  services  need 

-  to  Jre/prqvideS  ip-  a&jition-  to  that:  activity  to  meet  adequately 

tpte  needs  of  thp^arget  ^r&bypj. -\  -'J-  t         .  '• 

•       \l^^eview,^pbt^ntial  Service  or  assistance  which  your  £>IC 

tde  to  /part ieipain^s^o the thaiv  training  include:  ' 

v       • ..  ■ 


\ 


•4- 


outreach/recruitment 
intake  \ 

♦  orientatiqn      .    .  * 

_  assessment  * 

counseling  •  -  > 

employability  development  planning 
coaching 
transportation 
health  services  A 
family  services  , 

J.egal„aid  .         *  > 

job  development  •"*.,* 
placement 

foiiow-through  /' 

You  can  use  the  results  of  your  PIC's^  needs  assessment 
to  make  that  determination.  %It  should  be  made  for  eAch  project 
— which  your  PIC  .will  ^spo»so£^--  Xou  can  use~  the -form  xuv..±he — . 
following  page  or  pne  similar  to  it  to  assist  you  in  planning 
the  scope  and  services #f or  all  your  PIC 1 s  "participant 
services  projeqts.  »• 


:^:ipa] 
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PROJECT  SCOPE  PLANNING  WORKSHEET  ' 


TARGET  GWOUF 


^Description  No 


as 


DBSQRIP. 
MAJOR 

tER(S) 


t  ; 


PROJBCT 
OBJECTIVE 


PRIMARY 
TACTIC 


OTHfiR  SERVICES  NEEDED  TO  ACHIEVE  OBJECTIVES 
(enter  names) 


) 


ESTIMATED 

COST 


9 
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.C.     ALLOCATING  FjgftNCIAL  RESOURCES 

J.  ' 

After  all  projects  are  thoroughly  defined,  your  PIC's 
remaining  task  is*to  allocate  the  financial  resources 
necessary  to  accomplish  each  prpject.     As  you  will  recall, 
the  initial  decision  regarding  fund  all6cations  is  -made  when  .  . 
you  develop  your  strategic  plan.    At  that  time,. you  obligate  the 
appropriate  amount  of  dollars  that  you* want  to' spend  to  achieve 
each  g<fcal .    This  decision  is  checked  again  when  you  , set  your 
PIC 1 s • program  objectives. 

Now,  as* the  final  ?6t  prior  to ^developing  your  plan 
for  operation  of  the  PIC's  program,  you  break  out  the' amount 
"  to  be  expended  on  eac%  project  in  order  to  estimate  cost' 

feasibility  and  to  ensure  that  there  "are  sufficient  resources 
*  to  deliver  the  fuil  scope  of  program  servicesjthat  are*  planned.  ^ 

Estimating,  ycpur  costs  for  EGS  projects,  as  with  defining  * 
their  component  parts,  is  a  straight  line  process  of  deciding  what 
each  product  or  major  deliverabla  will  cost.     Estimates  on 
the  participant  side  arp  somewhat  more  complicated.  F3|etor-s 
that  you  /ha^e*  to  take  into  account  in  making  your  estimations 
on  participant  serving  projects  include: 

allowances 
wage  s 

frir^ge  benefits- 
training 

serviges  to  participants. 
*  • 

Types  of  costs  which  you  will"  need  to  consider  within 

these  categories  in' making  your  estimates  include: 

*  annual  or  "one-payment"     f^ed  costs 
'(e.g.,  in surance ,  equipment? 

*  monthly  or  continuing  fixjed  co&'ts 
(e.if. ,  rent,  telephone) 

.    •      continuing  fixed  costs  not  related 
to  service -levels  (e.g.;  personnel) 

*  continuing  variable  costs  related 
to'  service  lev^is* (e.g. ,  travel, 
allowances) 


to 


•  . ^one-payment"  variable  costs  related 
to  the  needs  of  individual  program 
clients  (e.g.,  supportive  services). 

Undoubtedly,  the  best  single  source  to  provide  you  with* 

* 

the  information  and. assistance  in  estimating  your  participant-, 
related  costs  is  your  CETA* Prime  Sponsor.      The  PS  has  a  wealth 
of  experience  an<}  v£^bably  the  necessary  data  to  enable  you 
to  project  your  costs  .accurately.  * 

If,  on  the  other  hand,  you  want  to  or  have  to  do  some  of 
your  own  cost  estimation,  the  instructions  whicfc  follow 
should  prove  helpful: 

•    Allowances  . 

for  each  project  and  all  projects  together: 

estimate  the  number  of  individuals  to  be 
served  who  can  qualify  for  allowances 
based  on  the  characteristics  of  the 
target  groups  and  on  the  qualification 
criteria. 

determine  the  minimum 'allowance  payment 
and  the  additional  provisions  for  depen-' 
dents.  .  * 


estimate  the  civerag£  number  of  weeks 
for  which  an  allowance  will  be  paid, 
based  on  the  duration  of  the  training 
program. 


The  format  below  can  be^ 
allowances. 


ed  for  calculating 


Format  for  Calculating  Allowances 


Project/Component 

t 

Basic  Training 
Allowances 


No.  Participants 
Qualifying 


Average  Basic  ^Q  Q^ 

Allowance  <\. 

M    .  Weeks 
Per  Week 


X  $ 


=  $ 


Allowances  f^r         No.  Participants         Number  Weeks     Amt.per    Avg.  No. 
Dependents-       -  Qualifying  of  Training       Depend,     of  Depend. 

  X  X  $   X    =  $ 


Meals,  Trans-  No.  Participants        Ho.  Weeks  Avg. Allow, 

portation, etc.  Qualifying  Particip.  for  this 

in  Trng.  Purpose 
per  -Week 

'    '  *  X  ,  X*  $  .  =  $ 


4        »       TOTAL  ALLOWANCES      *  $_ 
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*  •  Wages  and  Fringe  Benefits 

For  each  project  and  then  all  projects: 

-  determine  the  wages  to  be  paid  in 

each  of  tjie  specific  occupations 
included  in  the  employment  or'  train- 
m      ing  project  based  on  minimum  wage  #. 
levels  (i.e.,  the  higher  of  applicable^ 
federaj.  or  st^te  jfnd  local  standards) 
and  the  maximum  allowable  yearly  wage 
under  CETA . 

-  determine  the  "average  number  of  employ- 
ment or  training  positions  in  each 
occupation, -  based  on  project  objec- 
tives and  scope. 

-  estimate  the  average  number  of  months. 

'  each  participant  will  spehd , in  an  employ- 

ment, or  training  project. 

-  calculate  total  wacj.es  by  multiplying 
all  of  the  above  factors  and  for  all 
occup^ttipns. 

-  calculate  total  .fringe  -benefits  by 
multiplying  applicable  benefit  rates 
against  total  wages  and  sum  for  all 

~ occupations. 

The  , format beloW  can  be  used  to  calculate 
wage  and  fringe  benefits. 


•                      Format  for  Calculating  Wagss  and  Fringes  y 
Proi  ec  t/Componen  t  * 

 ,  s  =  %                         \  1  — 

(1) 

Ocbupat  ion 

(2) 

*  i 
• 

Number  of 
Positions 

C3) 

Avg.  Hours  of 
Employment  or 
Training 

(4) 
Average 
Hourly 
Wage 

(5)       .  \ 
Total  Wages  f&r> 
-  Occupation  t 
Cols.  2x3x4* 

(6) 

* 

Fringe 

Benefits 
 r 

- 

» 

r 

> 

• 

Total  Wages 
? 

Total 
Fringes 

? 

 ,  i  ■  •  ' 
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Training  • 

For  each  project,  and  then  all ♦ projects : ' 

-  determine  the  occupational  areas  for  . 
4,       training  and  the,  average  number  of 

trainees  in  each  area  based  on  the 
project,  objectives,  and  scope. 

-I  identify  any  applicable  state  education 
board  standards  -for  certification  of  the 
training  in  each  oTith6  occupational  areas 
The  standards  may  relate  to  qualifications 
•of  instructors,  minimum  periods  of  train- 
ing, minimum  and  maximum  class  sizes, and 
required  training  materials  and 'equipment . 

-  determine  project  administrative  costs*  - 
1         by  calculating,  'personnel ,  *f  abilities, 

and  equipment  Sosts  as  indicated  below: 

^     .  Compare  required  personnel ,  as 

determined  in  th^  prbject  o^aniza- 

*  tion  -and,  staff ing  plan,  to  tile 
laboj:  market,  in  terms  of  available 
personnel  arid  prevailing  wage  rates  -  - 
and  compensation  plans,  including 
employer  contributions  to  social  , 

.*  security  and  UI.  ' 

.  Compare  *  required  facilities  .and 
equipment  related  to  managing  trhe 
program  to  its  Availability  and 
to  the  prevailing  ^ents  and  costs. 

*  Compare  necessary  utilities  and 
maintenance  .services  against  pre- 
vailing rates. 

..  -  . 

.  Establish  the  permissible  indirect 
cost  rate  and  apply  it  against 
indirect  cost  base  items, 

-  identify  certified  training  •  institutions 
in  area  and  survey  them  to  obtaih  relative 

»  cost  estimates  to  verify  cost  Estimates. 

Services  tQ  Varticipanfcg  v 
For  each  project  and' then  all  projects: 

-  ftfffibto  the  procedure  as  described  above 
. for* calculating  personnel,  facilities 

and  equipment *c$sts  based  on  services 
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to  be  provided,  jiumber  of  persons 
receiving  each  service,  number  and 
types  of  personnel  needed  to  provide    ■  7 
services,  and  geographical  areas  to 
be  served  (i.'e./to  determine  the  need 
for  one  Central  center  vs.  several 
outlyirig  service  centers) . 

-  identify  special  services  which  may  be  y 
needed  by  a  few  clients  (e.g.,  eye- 
,    /     glasses,  dental, care) ,  estimate  the 
number  of  individuals  to  receive  the 
services,  and  determine  cost  based  on 
the  availability  o*f  such-  servAGes  in  t 
the  market.       *  , 

,  If ,  after  you  do  your^cost  estimation,  there  are  short- 
falls- or  overages  either  in  total  or  within  a  given  area, 
you  re-budget  and  adjust  your  program  objectives  and  project 
objectives  and  scope  accordingly.     After* fchis  activity  is 
completed,  you  are  in  a  position*  to  deyejbpP  jour  PIC's  plan 
for  program  operations.        #t  ; '  <>?  '  f 


,  Manpower  Administration  ffnow  Employment  and 
Training  Administration   [ETA]),  Manpower 
Program  Planning  Guide   (Washington,  D.C.: 
U.S.  Department  of  Labor,  April,  1974).  , 

Mayor's  Office  of^Manpower,  City  of  Chicago 
Planning  and  Evaluation  under  CETA  Wash-^ 
ington,  D.C.:  U.{s.  'Department  of  Latfcr, 
^January  1976) . 


\ 
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D.     DEVELOP  PROGRAM  OPERATIONS  PLAN 


Purpose:      To  estkhiish  a  detailed  plan  and  procedures 
for  ope^iting  your  PIC's  program. 


Steps 


1. 
2. 
3. 


Structure  Program  Delivery ' System 
Establish  Performance  Standards  • 
Secure  Necessary  Assistance  \  • 
Finalize,  and  Approve  Operations  Plan 

\ 


iii-.ife"  f,3J  • 


Step  I 


Structure  Program  Delivery  System 


The  purpose  of  this  step        to  formalize  the  mechanisms  , 
and  arrangements  for  delivery  of  the  full  scope  of  the 
program,  and  projects  which  .your  KEC  has  decided  upon. 
This  involves:    designing  specif i<j  projects?  .developing  * 
project  delivery  ;framework?  and  establishing  administrative 
infrastructure.  ,  ' 


Project  Resign  -  objeg^i 
ties  arfa  services  to  Td£ 
and  basic  organization  an 
the  project.  %^ 


e  accomplis't^d,  ro&jor  activJ 
action  steps^  to  be  taken, 
t  plan  for  delivering 


PIC  Delivery,  System  -  -overall  framework  for -providing  PIC 
training  and  development  (participant)  and  employment^ 
generating  projects. 

PIC  Administrative  Structure*-  arrangement  for  Coordinating 
and  managing  PIC's  projects. 


U  Knotrledge7C*pab1)1t1es  bating  £ 


/  Review  the  "information  provided  above.  Rate  your  PIC 's  knowledge  and 
^•capabilities  with  reference  to'  this  step  on  the  scales  below. 


Knowledge 
•  .Capabilities 


L 


_1_ 


± 


J 


•Excel-, 
lent 


Excel- 
'  lent 


Above 
Avg. 

X 


Above 
Avg. 


Aver- 
age 


Below 
Avg. 


Poor 


Aver- 
age 


Below 
Avg. 


Poor 


'Based  upon  your  .ratings,  decide  whether  you  need  to  read  the  content 
information  on  tnis-s-tep  which  foll&ws  immediately  or  proceed  to  the 
next  step. 

■'■   >•  J  '  •  ^ ^ ^™ 
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•  ^ME  PROCESS  NOTES/ ACTION  PLANNING  P&GE 

*  * 

NOTES   (Points  that  are  important  which  you  Wajit  to  remember) 

♦             *— •        •     ,           -  . 

• 

• 

• 

* , 

1    -  ' 

•    •    .               •  *  •    •    .  : 

w 

QUESTIONS   (Items  on  whick  ybu  are  unclear  or. want  to*  get  additional 
information)  ' 

* 

*  ■      -  * 

• 

ACTIONS   (Specific  steps  that  you  intend  to  take  to  implement  ,your  * 
P'lC's  PME  process' in  this  area)                        -  , 

Steps                          Resources/Persons,                 \Time  Frame 

f 

\ 

-    -  N 

/                                                         -                                                                   14  . 

..'   .  V 

>   •             -          '*  •  ' 

"                                                                             «      **  » 

> 

• 

■  ■          ■  .  -          1  ■   '  . 

V 
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In  this  step,,  your  PIC  makes  the  decisions  which  are 
probably  mtfst  critical  to  its  programmatic  success  or 
failure.     These  decisions  relate  to  how  you  £re  going  to  ' 
deliver  what  you'liave  decided  ufcon  as^your  program  for  the 
year.     Basic  considerations  to  be  factored  into  .your 
decision-making  at  this  point  include: 

•      whether  to  use  all,  fpart,  or  none  of 
the  prime?  Sponsor  delivery  system 


r 


how  muoh  direct  control  will  your  PIC 
want'  over  those  projects  whiGh  you 
sponsor  * 

•  ^hat  are  the  relative  tradeoffs  in 

%    terms  of  cost  and  credibility among; 
various  types  of  service  deliverers^ 

•  should  essential  services  be  delivered 

on  a  decentralized  qr  centralized  < 
basis,  y 

If  your' PIC  has  clearly  defined  its  own  mission  and 
function ,  then  many  Qf  the  answers  regarding  its  role  in 
the  delivery  system  will  be  readily  appsurent.     At  this  point, 
you  need  to  make  your  PIC's  role  even  more  precise  ajid  t9 
decide  what  *oles  others  are  to  play.     The  tasks  necessary* 
to  do  that  are  to: 

•  design  specific  projects  % 
%           •     develop  a  project , delivery  framework 

•  establish  administrative  infrastructure. 

'    ■  .  '  v  •  • 

A.     DESIGNING  THE  PROJECTS 

  \ 

The  first  order  of -business  in  establishing  your  PIC's 
delivery  system  is  designing  the  projects  which  are  to  be 
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initiated.*    If  your ^PIC  has  decided  to  run  all  or  certain 
projects  itself^  then"  a  great  amount  of  your  attention .and 
effort  will  go  into*this  area. 

Tke  prinicpaL  responsibility  here  is. to  decide  for  each 
project: 

•  the  major  activities  or  services  to  be  * 
•a'ccompjished  and  the  order  in  which  they 
are  to  be  accomplished 

•  the  specific  steps  or  actions  which  are 
to  be  taken  to v accomplish  each  activity 
or  service  -  including  resources  required 
^nd  time  frame  for,  accomplishment 


•       the  basic  organization  and  management  plan  ' 
'for  delivering . the  project. 

Your  PIC  can  use  the  general  formats  below  and  modi-fy 


as  appropriate  to  facilitate  its  general  pro ject  desdgn  process. 


Project  Design  Format^ 

>  ■» 

1 .     Project  Planning  Format 


PjTQject:             \  «  t 

< 

MAJOR  ACTIVITIES/SERVICES 

1  ■  ^  =**J 

'PERFORMANCE  STANDARDS  4 

2 .    Activity/Service  Planning  Forijiat 


PROJECT:     ,  * 

MAJOR  ACTIVITY/SERVICI 
i  — — 

Z:           C                 PERFORMANCE  STANDARDS: 

-jl.   ^ACTION  STEPS ' 

2/  RESOURCES/ASSIS. 
REQUIRED  • 

3,  TIME 
FRAME 

4,     PERSON  DAYS 
ALLOCATED 

* 

<  « 

• 
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3.     Management  Plan  Format 


ACTIVITY/TASKS 

STAFF  RESPONSIBLE 

%  OF  TIME  TO  ACTIVITY- 
i 

\ 

« 

• 

% 

r 

1 

• 

•  * 

Since,  at -this  time,  th'e  majority  of  PIC'S  contract  most- 
activities,  basic  principles  and  factors  related  to- effective 
project  design  .are  also  discussed  under  "Step  3.  Secure  Necessary 
Assistance,"  -which  follows.     The  same  principles  wh^dh  apply 
t<?  contracted  assistance  should  aPpi'Y  to  those  projects 
.sponsored  by  your  PIC  itself.  '  „  , 


B.     DEVELOPING  THE  PROJECT  DELIVERY  FRAMEWORK 

Regardless  of  whether  your  EIC  implements  all  its 
own  projeqts,  contracts  them,  or  is  somewhere  in  between, 
you  need ' to  establish  an  overall  framework  for  coordinating 
'and  integrating  your  PPC's  projects.     That  framework  will 
have  two  side§  —  the  training  and  development  or  partici- 
pant  side,  and  the'EGS  side. 


\ 


Training  and  Development  Side 

The  general  model  of  CETA  participant  client  flow  is 
indicated  in  the  chart  on  the' following  page. 
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CETA  Client  Plow  ttodel 


Outreach/ 
Recruitment 

4 

•< 

* 

f 

Intake/ 
Assessment 

t 

r 

Training  or 
Development 

*  • 

* 

.  Placement/ 
Termination! 

r 

Follow-up 

Participant 
Services  ^ 


Basic  questions  that  y6i*  can  ask  yourself  a*s  they  relate 
to  the  foregoing  model  and  your  pic's  projects  include: 

•  How  will  your  PIC  handle  its  outreach, 
recruitment,  intake,  and  assessment  of 

*  '         participants?    Through  a  central  source? 
Through  each  service  deliverer?  • 

•  What  provisions  will  be  made,   if  any,  to  "e 
meet  the  speoi^l  development  and  support  * 
service  needs  pf*  participants? 

•  Will  any  placement,  job  develdpment,  and 
program  termination  activities  be  under- 
taken? ' 
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•     Will  your  PIC  make  special . arrangements 

for  follow-up  assistance  to  pro.gram  ser-  p 
,  vices?    Yes?  No? 

* 

•    Your  -answers  to  -these  and  other  questions  which  you 
consider  important  will  determine  the^ shape  of  the  partici- 
pant service  side  of  your  PIC's  delivery  system.    Again,  as ^ 
in  estimating^ costs,  your  Prime  Sponsor  should  be  an  invalu- 
able  source  of  information  and  experience  regarding  service  ^ 
delivery*  arrangements  and,  in  many  cases,  a  potential  ser- 
vice deliverer  itself.    You  should  consult  with  the  Prime 

Sponsor  as  a  first  resort  to  help  you  plan  the  participant 

r 

side  of  your  program. 

*No  jnatter  what  specific  client  processing  arrangements 

you  decide  on,  there  are  some  basic  steps" that  should  be  taken 

to  allow  for  effective  operations  on  thd  participant  side  of 

your  PIC's  program.    These  steps  are  outlined  below: 
•*•*.« 
1.     Specify  for  each  project' the  sequence 
v      ,    *    in  which  participants  will  receive 
services. 

2;,   Establish  procedures  "and  mechanisms 

nece^sffiry  to  ensure  that  participants  • 
move  from  one  service  to  the  next 
wit^  taii*fmal  complication; 

3.  Define  in  writing  the  process  for 
coordination  amoncj  your  projects  and: 
services.  \ 

4.  Identify  the  total  -number -of  persons 
to  be  served  irt  your 'total  program, 
in  each  pro ject, '  and  the  number  to 
receive  specif ic 'services. 

5.  .Schedule  the  flow  of  participants 

through  your  delivery  system  to' 
determine  who  will"  bfe- receiving 
tt        '  what  types  of  services  at  what  time.    •  . 

6.  Balance  participant  levels  among 
services  throughout  the  program  to 
ensure  that  existing  service  capacity 
is  being  used  at  maximum  efficiency 

\  and  to  avoid  bottlenecks  or  under*- 
.   ,  enrollments. 
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The  actual  mechanics  for*  accomplishing  these  steps  are 
somewhat  technical.     However,  with  the  assistance  of  your 
Prime  Sponsor,  you  should  be  able  to.  structure  a  systenj  „ 
that  works  for  your  PIC,  / 

EGS  Sidle^  •  * 

To  structure  the  EGS  side  of  ypur  delivery  system,  you 
should  answer  questions  including:  " 

•  •  What' end  products  or  results  will*  be  » 

produced?    In  what;  time  period  or  by 
what  date?  *  _ 

•  What  interim  products  will  be  produced?* 
By  when?       •  ' 

n  •      What  major  activities  will,  be  accom- 
plished?   By  when? 

«        •      What  ^re  project  benchmarks?    When  will 
they  occur? 


What  are'  the  major  tasks  or  steps  nedes-  • 
s'ary  to  complete  ea<ph  project?  WhatVLs 
their  timeline?. 


Answering  these  questions  for  all  EGS  projects  will 
enable  ydu  to  develop  a  time-phased  product/activity  schedule 
which  can  be  used  to  direct  the  implementation  of, your  PIOfs 
EGS  projects  over  the  course  of  the1 year.     A  sample  format  for 

i 

such  a  schedule  is  presented  below. 


r  '  PROJECT/ Activities 


8 


10 


11 


12 


Activity  1 
Activity  2 


A  r  major  benchmark  ' 
Q  •=  "deliverable  product 
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C.     ESTABLISHING  THE  ADMINISTRATIVE  STRUCTURE 

'  V 

t  M  " 

A$  the  last  task  .in  developing  its  program  delivery  sys- 
tem, your  PIC  sfeould,  design  and  put  in£o  place  an  administrative 
infrastructure  which  complements  and  supports,  your  decision^ 
regarding  participant  and  EGS  project  delivery.     To  dc5  this, 

you  must  review  and/or  decide:  ,  *  • 

•  «  * 

•  "  ' What- projects  the  PIC  will  run  itself  # 

and\the  staffing  required 

v 

#A    How  to  direct,  coordinate,  and  cbntrol  y 
all  projects  to  be  initiated.  •    m  • 

In  addition,  you  should  assess  your  administrative  and 

management  support  requirements,  particularly  in  the  areas 

of:  '      '  ,  '  '. 

,  •'      Financial  Management 

»  ' 

•  pata  Collection  and  Reporting 
\      Planning,  Management  and  Evaluation - 

•  .    Personnel  and  Staff  Activities 
%      Procurement*  ancT Maintenance  *  K  1 

•  ; x  Planning  * 

•  Correspondence  and  Communication  Control 

•  , Public  Information 

«      Grants  Administration  »  '  • 

•  Legal .  *  \ 


The* decisions  that  you  reach  here  'will  determine  the 
4ay-tQ-day  administrative  organization  and  the  staffing 
needs  for  your  Pit.     Once  you  -have  made  these  decisions, 
you  shpuldVjesicJn  an  initial  organizational  and  staffing 
plan  for  yo^  PIC.     That*  plan  should  detail  the:- 

•  structural  design  of  youfc  PIC's* 
organization  (this  design\jiay  txe 
for  flill  time  PIC  staff,  'CETA  staff, 
shfared- staff ,  mixed  staff,,  or  any 
other  arrangement'  you\deem  appropriate) 

•  '  primacy  .functions*  to  be  performed 
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level  of  staffing  needed  to  support 

the  program  r  -  *  *  , 

•      job  descriptions  .for  staffs  x 

Finally;  as  with  .your "PIC 1  s  projects,  as,  a  last  step  'to 
ensuring  your  administrative  plan  can.be  implemented,  you 
should  calculate  ^ur  administrative  costs  .based  upon  the 
decisions  you  have,  made.    Again,  factors  to  take  into 
account  iAcludl  personnel,  facilities,  and  equipment  co&ts. 


Manpower  Administration  (now  Employment 
and  Training  Administration  [ETA]),  . 
Manpower,  program  Planning  Guide  (Washing- 
ton,  D.C.*:  U.S.  E 
1974)  . 


Department ,of  Labor,  April, 


Manpower  Administration  (now  Employment 
arid  Training  Administration  [ETA]), 
Program  Activities  and  Services  g«ide 
("Washington,  D.C,  :  UVS/  Department  of- Labor, 
*April,'  1974)  . 
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Establish  Performance  Standards 


The- purpose  of  this  stfep  i*s  t9  set  the' performance 
standards  for  your  PIC.     Standard^  should  be  set  at  the 

program, project/  and  component  levels. 

'  *        ♦         *  '  •  • 


Performance  Standards'  -  statements  of  measurable.  Tesults. 
to  be  achieved  and  used  to  ascertain  success  <^  effec- 
tiveness •  , 

•  Performance  Indicators  -  measurable  factors  ,(e.g.,  entered 
'  employment  raTe,  number  of  jobs  retained  for  CETA-eligible 

participants,  as  a  result 'of  industrial  revenue  bonds) 
indicating  effectiveness  wi^in  a*  given  area  on  which  it 
'  <f"ik^amportant  to  set*  objectives  and  standards.    '\  ^ 

•  Terminal  Ind icatqr s  -  measures  used  to  assess  the  end  results 
or  outcomes  of  a  pro ject, < component /  or  program. 


Interim  Indicators  -  measufes  used  t<?  assess  the  ongoing 
rformanqe  of  PIC's  projects,  components ,  and  program. 


\ 


|P  l<r  Krtbwledge/tapibffltlcs  Ratify 

Review  tlfe  information  prbvided  abpve^  Rate  y°ur  PIC'S  knowledge  and 
capabilities  with  reference  to  this  step  on  the  scales "below. 


•  Knowledge 

•  Capabilities 


1  

•  1 

i 

I 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

1 

_i  _ 

• 

1 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg.  , 

Based  upon  your  ratings,  decide  whether  you  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the. 


next , step. 
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0'     PME  PROCESS  NOTES/ ACTION  PLANNING"  PAGE 


7 


NQTEpT(Po:Lnts  that  aye  important  which  you  .want  to  remember > 


/ 


QUESTIONS   (Items  on  which  you  are  unclear for 
]       information)  n 


nt  to  get  additional 


r 


\ 


y 


ACTIONS   (Specific  steps  that,  you  intend  to  take  to  implement  your 
.  |FIC's  PME  process  1jAj^s  ^rea) 

Steps  "       R  e^p^Fe  s  /  Per  sons  ^  •    Time  Frame 
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INTRODUCTION,  < 

.    v  ,  : 

After  you  haye  structured  your  program  delivery  system, 

your  PIC's  next  step  Is  to*  establish  its  performance  stan- 

aards  for  the  fiscal  year.^  Performance  standards  are  measurable 

description^of \  what  constitutes  a  job  well  done  for  your  PIC. 

'  Performance  standards  are  useful  in  all  phases  of  the 

PIC's  program.  They: 

'  1  ,  'I  ' 

•  help  you  t'o  target  resources  and 

#  activities  during  the  planing 
process  ^ —  • 

•  serve  as  measures  which  can  be  used 
on  ean  ongoing  b#sis  to  assess  rela-  % 

♦  tive  progress  atnd  performance 

•  • 

•  -provide  a  meartfe^  for^  making  an  obj.ec- 
.  tive  assessmeat*  of  PJC  program  per-  i 

formance. 

*<  *  .  V. 

If  you  fcset  program  objectives  and  project  objectives 

J    as  describe^  in  Stage  C,  ^Establish  Program  Scoped"  then  you 
already  have  the* primary  input  to  use  in  establishing  your 
PIC's  performance  standards.    To  make  these  objectives  intb 
standards',  you  adjust  th'em  to  reflect  any  new  insights  which 

*■    -you  may  have  attained  as  a  result  of*  structuring  your  delivery 
system;  or  to  account  foi"  changed  conditions  in  the  local  v 
labor  market  or  economy  since  your  initial  objectives  were 
drafted.    These  standards  will  then  relate  directly  to  your 
.PIQ's  goals  and  tactics.     They  should  be  sufficient  for  program 
management  and  review  purposes.     In  addition,  at  this  point,  j 
however,  you  may  want  to: 

m\    refine  the  standards  you  have  developed 

•  create  additional  overall  or* com- 
poifent  standards 

•  develop  more  particular  standards  , 
,  *    for  each  project.  -    -  * 
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•  \     ,  ... ;     **   ■  -     j  •  - 

The  ctfhcejgt  which  is'  .crl€ic»l  to  the  development  of  stan- 
dards is  "indicators.*    Indicators  tell  you  what  is  to  be  .  ♦ 
Measured.     They~  are  me^surablfe'  factors  indicating  effective^ 
.  ness  which  relate  zo  a-Tsertain  area  of  activity..    By  establishing 
a  specific  acceptable  value  for  each  indicator  that  your  PIC 
selects,  you  can  crfeate,  a  program  performance  standard. 

The  discussion,; which  follows  ^describes  various  indicators' 
that  c$n  be  usedj  to'^eyelop  standards  for  the  participant/         .  ► 
employment  generating,  and  administrative  aspects  of  your 
program.  <       *  -  - 

«    x-  *  . 

A.     DEVELOPING  TRAINING  AND  DEVELOPMENT 
(PARTICIPANT  SERVICE)  STANDARDS 


The  Department  of  Labor  is  presently  developing  national 
performance  standards  for  application  to  all  CETA  Prime  Sponsors, 
Indicators  which  have  been  selected  as  appropriate  for 
Titles  II-B  and  II-b--those  CETA  program  components  closest 
to  Title  VII's  in  intent  and  'structure — are: 

•  entesed^0mploymant  rate 

•  ,  positive,  .termination'  rate  * 

•  average  wage  at  ^placement 

•  cost* per  entered  employment 

:  •      cost  per  positive  termination. 

YQur  PIC  may  fcant  to  establish  its  own'  standards  for  each1 
of  these  indicators^     Additional  overall  participant ^.ndicators  ' 
for  Which  your  PIC  might  develop  standardf,  or  objectives  include: 

4  * 

•  .   indirect  "placement  rate 

*  •  private  sector  placement 

•  cost"  per  indirect  placement 

•  total  cumulative  enrollment 
»  totarl  current  enrollment 

•  tdt^l  accrued  expenditures. 
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Examples  of  pajfticipajnt-related  performance  standards  estab-  - 
lished  by  'two  'PKs  follow.  '  .  * 


PIC  A  -  Performance  Standards  for  All  Classroom 

rTfrainigg  Projects  '  ~ 


Assessment  Category 

Indicator 

Standard 

TERMINATION".  - 

Positive  Termination  Rate 
Entered  Employment 

.  7*1% 
68.0% 

COSTS 

Cost  per  Positive  Termination 
Cost  per  Entered!"  Employment 

$4,856.00 
5,511.00 

RESOURCE  USE  f 

j 

*  • 

To^al  Cumulative  Enrollment 
Total  Current  Enrollment 

w 

Total  Accrued  Expenditures 

170 
39 

$490,468.00  • 

. ...    '      "     .''  N 

PIC  B  -  Performance  Standards  fo*7 Various  Projects 

T 

Projects 

Indicator 

Standard 

OJT 

«  • 

!  *  1  "  ~  

 i  Tt{*  2=— 

Entered  Employment 

Satisfactory  Completion  and 
Retention  by  Emplox^r 

;  *  ■  '  1 

70% 
66% 

CLASSROOM 
TRAINING 

Entered  Employment 

-%  Entered  Employment  in 
■  Private  Sector , Jobs'" 

P6sitive  Termination^ 

70% 
90% 

75% 

BARTICIPANT 
PLACEMENT 

%  of.  Non-Administrative  Funds 
-  to  be  Expended  as  Transfer 
Payments  to  Participants 

70% 

,  minimum 

*> 


-  i  • 


Other  indicatory  for  which  your  PIC  could  develop  % 
standards,  ^as  they  relate  to  more  specific  part^Qf  the 
participant  sidje  of  your  program,   include':  •  * 


Component  , 
Outreach  and  .Recruitment 

Intake 

Vocational  Exploration 
Upgrading  , 

Retraining 


Job  Development 


Possible  Indicator 

%  of  eligible  referrals  irt  , 
-relation  to  planned  referrals 

%  of  applicants  confirmed 
eligible  * 

%  of  successful  completers, 

Number  or  %  of  participants 
promoted 

Number  or  %*  of  paxticiparits 
securing  new  positions  with 
salary*  equal  or  greater  to 
previous  wages 

Number  of  jobs  developed 


f 


B.     DEVELOPING  EMPLOYMENT  G&ERATING. 


SERVICE  STANDARDS 


'indicators  which,  your  PIC  might  use  in  developing  stan- 
dards for  employment  generating  services  are  presented  below 
'by  the  major  category  to  which  they  relate: 


Category 
Linkages 


Possible  Indicators 

•  Number  of  formal  or  informal 
agreeijien£s— ior  loaned  staff, 
use  of  -facilities,  and  other 
resources  * 

•  «  Number  <\f  formal  or  informal 

agreements  completed  with: 

-  business  agencies,  * 

-  community  groups, 

-  unions, 

-  schools  < 

-  others 

gt  Number  of  business  schools 
involved  in  joint  training 
programs  with  PIC 
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Category 

Economic  a'nd  Community 
Development 


Possible  Indicators* 


Incentive  Program 


Marketing 


•  Number  of-  jobs  retained  (or  -added)' 

•  locally  as  a  result  6f  issuing 
.industrial  revenue  bonds  to 

companies  i&bout  to  close  or 
relocate*. 

•  N\pbe(r  of  new  jobs  created  ear- 
marked for  PIC  participants*: 

•  Number  of  new  jobs  filled  by 
PIC  participants.  m 

m  Number  of  new  jobs  for  PIC  par- 
ticipants developed  as  a  result 
of  attracting  new  business.   "  * 

•  Number  of  program  participants 
successful  in  getting  funding 
and  training  for  opening  a  • 
small  business. 

•  Number  of  new  small  businesses  , 
established  in  target  neigh- 
borhoods. 

•  Number  of  workers   (or  program 
'participants)  served  by  PIC- 
initiated  transportation  program 

•  Number  of  program  participants 
who . have  found  permanent  > 
employment  in  PIC-initiated 
day  care  centers. 

•  Nflfober  ofi  program  participants  ^ 
placed  if  jobs  who  are  alsto 
using  PIC-initiated  day  dare 
centers^.  ^ 

•  Number  of ■ employers  placing 
clients,. 

•  Number  of  clients  placed  through 
use  of  TJTC  or  other  incentive... 

•  TJTC(  positions  generated  through  . 
seminars  arid/or  direct  sales 
efforts. 

•  Increase  in  number  or  %  of  businesses 
participating  in  the  hiring  j?r 
training  of  PIC  participants . 

f  Number  of  fcelephbne  .and/or  -  letter  ^ 
inquiries  from  employers  as 
result  of  specific  advertising  * 

4   effort.  a 
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Category  % 
Marketing  (coot.) 


Labor  .Market  Analysis 


Possible  indicators 

•  Amount  of  private  sector  funds 
contributed.  « 

•  Number  of  businesses  who  have 
agreed  to  enroll  supervisors  in 

' PIC-sponsored  training  in  how 
to  assimilate  C&TA'  clienjts. 

•  Response  rate  to  local  employer 
survey. 

•  Utility  of  information  in  iden- 
tifying potential  job  oppor- 
tunities, hiring  requirements, 
wage  structure. 


r. 


.     .  C.     DEVELOPING  ADMINISTRATIVE  STANDARDS  r 

*  *         «  * 
.The  last  major  area  in  which  your  PIC  might  want  to'estab- 

lish  program  performance  standards  is  on  the  general  administra- 

tion  of  the  PIC  and  PSIP  itself.     Using  indicators  such  a*: 

•  ^representativeness  of  Council 

•  average  attendance  at-  Council  meetings 

•  description  of  operating  policy  and  procedure,  „ 
'    •      adequacy  and 'accuracy  of  program  accounting 

•  frequency  and  quality,  of  reports 

•  level  of  participation  by  Council, 

your  PIC  can- create  internal  performance  standards  which  will 
contribute  to  overall  program  Effectiveness.  . 


•  D.     DEVELOPING  OTHER  PROJECT  STANDARDS 

•  X'      '     r  '  - 
The ^indicators  and  standards  described  under  the  foregoing 

headings  ar^  primarily  "tlhaaiflfl"  in  nature.     That  is,  they 

relate  to  end  results  ^r  outcomes.    Your  PIC's  projects,  h<3Wever, 

are  a  process..   They  consist  eitjier  of  a  set  of  services  or  • 

major  activities- which  are  either  ongoing  or  to  be  accomplished 

in  a  sequential  fashion: 
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.}***-  .      '.V-  ' 

Therefore,  it  is# important  that  for  each ^project .your 
PIC  also  evaluate  the  need  for  "interim"  or;  "concurrent" 


/ 


j  indicators  and  standards.     In  those  Areas  where  you  eel 
#  that  you  should  not  wait  for  end  results, ^you  should  also 
develop  interim  indicators  (e.cj.,  costs  per  month -acccyrding 

'to  plan;  number  of  participants  processed,  ref>orts  submitted 
as  planned;  contacts  made;  monthly^Km-pokitive „ termina- 
tions; benohraarks  met)     -  These  indicators  ancl  standards  serve 
as  the  basic  measures  to  monitor  your  PIC's  program  on  a 
regular  basis. 


rM 

•8 


CONCLUSION 


V 


In  conclusion,  a  caution  ^.s  in  TJrder.  As  -the, foregoing 
discussion  suggests,  it  is  possible  to  establish  performance 
^standards .for  almost  everything  and  anything.    You  should  \ 
not  attempt  to  set  performance  standards^  f or  every  area  of 
PIC  activity.     Rather,  yo,ur  PIC  should  restrict  its  program 
standaircte  to  those'  areas  and}  indicators  which  you  feel  are 
key  or 'critical  to  assessing 'your  PIC's  program  performance. 


Employment  and  Training  Administration, 
CETA  Federal  Representative  Training  Program: 
Systems  and  Management  Analysis  Skills  Block 
(Washington,  D.C. :  Nat  1 


rational  Training  Center, 


ETA# .1981)  . 


Manppwer  Administration   (now  Employment  and 
Training  Administration  [ETA] ) ,  Manpower 
Program  Planning' Guide  (Washington,  D.C.:* 
U.S.  Department  of  Labor,  •  April,  1974).  m 

Manpower  Administration  (now  Employment- and  Training  Adminis- 
tration  [ETA] K#  Program  Assessment  Guide  (Washington,  D.C.  : 
U.S  Department  of  Labor,  April,  1974)  .• 
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Secure  Necessary  Assistance 


/ 

The  purpose'  of  this  step  is  to  secure  assistance  or  services 
for  those  projects  which  the  PIC  does  not  intend  to  deliver 
or  provide  itself.     The  step  involves:     creating  list(sO 
of  potential  service  providers;  developing  and  issuing 
RFP's;  reviewing  and  selecting  service  providers. 


T 


Service  Providers  -  government  agencies,  community  based 
organizations ,  businesses,  and  other  contractors  who  furnish 
or  might  provide  assistance  to  PIC.  — 

•  Request  for  Proposal  (RFP)   - ^solicitation  used  to  inform 
potential  service  providers   (contractors  and  subgrantees) 
about  the  projects  which  PIC  intends  to  fund.  * 

^Proposal  Review  Criteria  -  objective  factors  which  are 
employed  to  review  responses  to  Request  for  Proposals. 


g(  1(   Know1ed9e/q*pab11U1gs  Rating  £Z>- 

Review  the*  information  provided  above. \  Rate  your  PIC's  k/iowledge  and 
capabilities  with  reference  to  this  step  "on  the  scales  below. 


•  Knowledge 

•  Capabilities 


L 


Excel- 
lent 


Excel- 
lent 


Above 
Avg. 


Above 
Avg. 


Aver- 


Aver- 
age 


Below 

hvq>* 


Below 
Avg. 


Poor 


Poor 


Based  upon  your  ratings,  decide  whether  you  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the 
jiext  step. 
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PME  PROCESS  NOTES/ ACTION  PLANNING • PAGE . 


ty)TES(PoiTit^that  are  important  which  you  want  to  remember) 


JETSTION£  (Items  on  which  you  are  unclear  or  want  to  get  additional 
~    information)  ,  0  C 


-j 


ACTION^(SpecTTic  steps  that  you  intend  to  take  to  implement  your 
*  1  PIC's  PME  process  ipi  this  area)  r 

'   Time  Frame 


Steps 


Resources/Persons 
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'*     \  INTRODUCTION 

The  extent  tya  which  your  PIC  has  to  secure  outside 
assistance  will  /be  determined  by.  the  basic  approach  that 
'it  takes  to  delivering  services'.     If  your  PIC  needs  out- 
side assistance,  the  process  ^iat  your  PI£  should  follow 
to  obtain  that  assistance  is  to:  , 

•_£reate  list(s)  of  potential  service 
1  •  providers 

r 

•  Develop  and  issue  RFPs 

•  Review  and  s*elect  service  providers^ 

This  process  is  -described  in  the  pages  which  foliow. 


A.     CREATING  LIST(S)   OF  SERVICE  PROVIDERS^ 

A  neces^jkry  pre-step  to  securing  a^sistanc^^a  developing 
a  list  of  potential  service  providers  for  each  project  which 
is  to  be  contacted'  out,    ^our  PIC  can  draw  these  lipts,  in 
part,  from  the "inventory  maintained  by  the  CETA  Prime  Sponsor. 
In  fact,  the  Prime  Sponsor  might  be  among  your  list  of  poten- 
tial  service  providers. 

The  Prime  Sponsor's  inventory  will  probably  consist 
mainly  of  agencies  or  entities  -which  currently  provide  or 
have*  provided  serviced  to  CETA.     As  part  of  your  responsi- 
bility to  increase  the  private  sector's  involvement  in  CEtA 
programs,  you  should  ^^and  the  list  to  include  business   v  ,  ' 
organizations  with  the  capability  to  provid^_services  which 
are  the  same  or  similar  in  nature  to  those  desir5at>sYc 
should  attempt  to  make  your  potential  "biddei^"  list(s)  as 
coihplete  as  possible,  "since  the  list  determines  what  organiza- 
tions will* receive  Requests  for  Proposals  issued  by  the  PIC. 

f 
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The  more  comprehensive  and  targeted  that  your  lists 
arje  for  each  project  in  terms  bf  designating  .all  potential 
deliverers  with  the  requisite  capabilities  to  provide  a 
service,  the  better,  the  chance  that  you  will  get  the  best  . 
deliverer  possible  for  the  service  in  question*^ 
•  * 

B.  '   DEVELOPING  AND  ISSUING  RFPS  ^ 

A  Request  for  Proposal   (RFP)   is  a  solicitation  usedtto 
'infprm  potential  contractor's  and  subgrantees  about  projects  *% 
which  your  PIC  contemplates  funding. '  Your  PIC?  should  develop 
and  issu*  a  Rfequest  for  Proposal  for  eachvproject  to  be 
funded  through  the  PIC,  but  to  be  operated  by  an  external 
agent.  \ 

In  fact,,  f^rom  a  management  standpoint,  it  might  be 
desirable  for  your^PIC  to  request  proposals  even  for  those 
projects  which  are. to  be  run  in-house.     This  is  frequently 
done  in  the  private  sector  in  businesses  which  work  o™toe 
"cost  center"  concept.     When- a  division  has  to  secure  fecial 
services  (e.g.,  data  processing,  training  and  development)  • 
from  outside  of  its  area,  it  asks  the  company's  service  arm 
to  prepare  a  proposal  describing  its  approach  to  providing 

« 

the  service  and  the  associated  cost.     This  enables,  the 
buying  division  to  obtain  exactly  the  type  of  service  it 
wants  and  to  obtain  the  cost  for  provision  of  this  service. 
Obviously,  this  concept  is  transferable  to.  the  PIC  setting 
in  which  there  is  such  a  wide  variety  of  service  assistance 
(EGS  and  training  development)  which  has  to  be  secured  and 
provided  in  the  most  cost  effective  and  efficient  manner. 

For  the  most  part,  however,  you  will . probably  be  „ 
developing  RFPs  to  secure  assistance  from  outside  of  the  PIC. 
The  amount  o^.  detail  in  eacli  RFP  will  vary  from  project  to 
project,  depending  upon  the  experience        the  recipients^ 
^rfd  whether  or  not  competitive  procurements  are  being  taken. 

At  a  minimum,  however,  each  RFP  sho'uld  corisist  bf: 


information  regarding  the  PIC  and  PSIP 
objectives  ^or  the  project  < 
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detailed  statement  of  scope  of  work  " 
(assistance/services)*  desired 

performance  measures  'and  standards  *'ta 
be  used  in  assessing  projects  /  -*  T 

performance  levels  of  prior  programs 
or  projects 

criteria  by  whictv  the  RFP  will  be  evaluated 
(see  next  heading  in  this  section) 

performance  schedule 

reporting  and  management  requirements 

description  of  the  procurement  policies 
and  regulations  of  the  PIC 

Additional  information  on  the  contents  and  how  .to  prepaTe 
an  *RFP .follows.  , 

Project  Title.  *Each  RFP  should  have  a  brief 
_      title  which  describes  the  project.     If  the 
project  is.  participant  related,  this  section 
could  also  include  summary  information  -'the 
total  number  of  slots  or  the  total  number  of 
persons  to  be  served,     if  your  project  is  * 
EGS  related,  this  sectior>*may  furnish  a 
♦one-sentence  description  of  the  specific  * 
assistance  to  be  provided. 

If  competiti^  fcids  are  "to  be  solicited,  this 
should  be  stated  in  a  note  below  the  titl>5, 
together  with  the  closing  date  for  receipt 
.  of  bids..  $ 

Program  Information.     In  this  section,  give       *  ^ 
enough  background  description  on  youjr  PIC's 
program  to  inform  a  potential,  contractor  or 
subagent  of  your  broad  concerns  and  perspec- 

•  tives.  Include  such  items  as  the  purpose 
statement  from  the  CETA  legislation,  your 
PIC's  mission  as* you  Have  devel6ped  it,  and 

.  selected  goals  from  your  strategic  plan. 
Because  of  the  general-  content  of  this  sec- 
tion, it  may  be  desirable  to  use  the  same 
statement  in  each  of  the  RFPs  developed 
during  a  planning  period. 


Project  Objectives;'    In*  this  sect'ion,  set  in 
context  the  particular  services  to  be  pro- 
vided by  the^ project  under  discussion. 


9 
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Describe  the  project's  specif ic^.pbjectives 
and  the  barriers  it  4s  intended  to  overcome. 
.Give  an  account  of  tlie,  obstacles  at  which  the 
project  is  aimed.    Fpr  sQme  RFPs,  it  may  be 
appropriate  to  set  out  the  .assumptions  under- 
lying the  service^. approaches  selected. 

Scope  J5f  tfork.     In  £his  section,  you  describe 
the  specific  sub-objectives  of* the >pro ject/ 
the  activities  to  be  performed ,  and/or  the 
services  to  be  provided.    Objectives  should 
be  stated  specifically  enough  sd  that  measures 
to  the "extent  to  which  they* are  achieved  can  ' 
be  clearly  defined.    Activities  or  services 
to  be  provided^shofclcTbe  described  ip  terms 
of  thfeir  major  -components.     Fdfcr  projects 
which  involve  several  components,  you  should 
/    stat^' objectives  and  describe  each¥.separately.  - 

Once  the-^bmponents  havei  been  established, 
expl^ift  for,  each  suph  element  what  has  been 
determined  regarding  the  services  to  be 
K   offered  and  the  operations,  organization,  /' 
•^management  and  administration  necessary  to  \\ 
implement ~thd(£port ion  of  the  project. 

s  * 

Organize  the  material  of  'this  section  under 

the  relevant  »ser.vice  headings .     Describe  any 

aspects  of  the  organization  which  cannot  be 

attributed  to  a  single  component  separately 

under  all  components fc to  which  it  applies 

and  al£o"  under  the  heading  of  "General 

Administration . H 

Finally/  if  this  is  a  participaht^-serving  * 
'project*  this  section ^should  pres^njr informa- 
tion ontehe  relevant  characteristic  of  the 
target . groups  to  be  served,  by ' the  project. 
It  should  include  a  statement  of  the  manner 
in  which,,  and  the  efttent  to'  which  the  project 
will  serve  the  areafs  priority  target  groups. 

.  / 

Performance  iMeasures  apd j^luataon  Criteria. 
In  this  section,  you  should  describe,  (a)  the 
^measures  which  will  be  used  to  monitor;  and 
evaluate  the  overall  performance  of  the  project 
and  (b)  the  evaluation  criteria  to  be  applied 
to  the  assessment  of  each  component  pf  the 
^    project.  '  Your  .evaluation  criteria  and  measures 
sjiould  include  the^performknce  standards  you 
*  have  set  fop  the  projects,  activities,  and  the 
formal  .comparisons  you  intend  to  make. 

-  314  . 
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In  addition,  in  this  section  you  should  re-' 
quest  that  the  potential  contractors  and 
subagents  submit  additional  performance 
.    measures  which  they  feel  appropriate.  In 
general,  you  should  list  separately  the  per- 
formance measures  which  apply  to  the  project 
and  then  those  which  relate  to  each  com- 
ponent. ^     %  4 

?  . 

As  a  caution,  you  should  attempt  to  keep 
the  number  of tperf ormance  measures  to  a 
minimum  consistent  with  validti-ty  (three  per 
/  component  is  a  reasonable  maximum) . 

Performance  Scheduik^  In  this  section,  you  •  - 

sBoWTncIudTlrtJaT^has  been  determined  on 
If  sojeduling  of  major  activities  and  the 
dates^fSn:  attaining ;  specified  achievement 
levels.  \Specify,  where  possible,  the  input, 
^service  uhdt  and-  output  measures  which  should 
be -used  by)the  potential  contractor  or  sub- 
agent  in  developing  the  performance  schedule*  • 
•  •*>  -C°«*ained  in  its  proposal.     Indicate  the 
T   length  of  the  periods  or  the  .dates -on  which' 
planned  achievement  levels  should  be  based. 
The  planned  achievement  levels  may  have  to 

•  be  specified  separately  for  each  component 
and  the  measures  should  correspond  directly 
«>  the  designated  performance  measures. 

v  ' 

Management  and  Reporting  Requirements.  This 
section  snould  describe  all  reports  which 
are  required  and  the  management  and  adminis- 
trative procedures  to -which  the  contractor 
will  Ije  required  to  adhere. 

Policies  and  Regulations/of  the  PIC  'in 
,  tnis  section,  you  should  detail  all  pertinent 
information  on  contracting  procedures  of  your 

•  PIC.     This. should  include  the- type  of  con- 
tract  to  be  awarded,  factors  related  to 

maximum  allowable"  compensation  for  ser- 
vices rendered,  if  applicable,  and  the  ' 
method  of  payment  for  the  contract.  ^ 

Structuring  a  .sound  RFP  is  an  important  step  in  the 
process  of  securing  necessary  assistance.     In  most  instanc 
the  quality  of  the  RFP  will  influence  -greatly  the  quality 
of  responses  which  you  receive  to  it. 
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C.     REVIEWING  AND  SELECTING  ' 
SERVICE  PROVIDERS" 

 \  • 

Once  you  have  received  responses- to »your  RFP,  the  next 
step  is  to  review  that  RPP.  to  determine  which  proposer  to 
select.  .Although  the  actual  process  of  reviewing  and 
selecting  service  providers  wAl  vary  considerably  from  - 
PIC  to  PIC,  it  is  critical  that  the  Council  take  a  leadership 
role  in  determining  who  service  providers  will  be^.     In  this 
regard,  it  is  probably  advisable  to  have  a  subcommittee 
or  ad  hoc  committee  to  review  and  rate  proposals. 

General  factors  which, should  be  taken  into  account  in 
conducting  that  rating  include  the; 

proposer's  understanding  of  the  scope  of  work 
sufficiency  of  the  content  and  work  description 
organizational  capabilities 
related  experience  c 
relative  costs  .for  delivery  of  services 
in  comparison  to  other  proposers. 

) 

Ti^e-VII  Proposal  Review  Criteria 

•  An  example  of  a  set  of  proposal  review  criteria 
developed  by  a  QPC  is  provided  below: 

"  program  Impact/Outcome      Potential  contribution 
of  the  program  toward  /he  resolution  of  the  x 
problem!  experienced  D*the  eligible  population, 
•     and  t^f  program's  potential  contribution  toward 

the  attainment  of  PIC  goals.  •  . 

•  4kll  the  proposed  program  result  in 

placements  in  the  private  sector  at 
a  rate  acceptable  to  the  PIC?.  . 

•  will  the  proposed  program- result  *«i 
proportional  placement  in  :obs  for  all 
significaAt~segments?  ^ 

\ 

•  Will  the  propo'sed  program  facilitate 

-    and  ^sult.in  retention  of  participants 
by  employers  after  placement? 

'  -  3lQ 
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Will  the  proposed  graining  and/br  ser- 
vices enhan*c^fc the  upward  mobility  and 
promotional  opportunities  o"f  .partici- 
pants?   :     •  . 


Program  Design      Potential  success  of  the 
program  design  in-  the  attornment  of  the 
described  outcpmes/itnpact 

•  Does  the  proposed  .program  satisfy  one 
or  more  of  tfre*c*iteria  foe  projects 
to  he  funded  under  Title  VII  within 
the-  PIC?     .  .J  • 

•  Does  the  proposal  .contain  a  fe&sible 
design  and  rationale  describing  who 
wil\  do  what'#  when^,  where  and  how 
(i.e.,  program  activities  and 'partici- 
pant *  flowt? . , 

•  •    If  the  cooperation  of  other  organiza- 

tions is  necessairy  to  successfully 
implement  the  program ,  do$s  the  pro- 
posal dQsppibe  the  coordination  with 
,     those  other  organizations? 

•  Does, the  proposal  describe  adfequat# 
and  potentially  successful  procedures 

*   for  outreach,  recruitment  and  assess- 
ment of  participants* that  will  ensure 
enrollment  of  significant  "segments  at 
.the  propose^  rates? 


Do^s  the  proposal  adequately  describe 
training ifpr  occupations  which  are  in 
dengftid  an<a  explain  how  persons  trained 
will  pe  su'ccessf uHj^-placed  J.nto  such  <^ 
jobs?*   *  *    •  ■ 

Does  the  proposal  adequately  describe 
how  the  offeror  -Will  provide  the  support  . 
necessary. to  sustain  participants  through- 
out the  training' !£n  order *to  reduce  ' 
attrition?       ~  ~* 


Program  Cost  -  Haw  do  the  following  cost 
factors  compare  with  competing  proposals: 

«       Cost  per  participant  enrolled  (totkl 
program  cost  minus^ income  transferred 
to'  participants)  .  (  i) 
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•  Cost  per  placement  (total  program  cost 
minus  income  transferred  to  participants) . 

•  Does  the  proposed  adequately  explain 

-and  justify  the  costs  of  the  prtfgram?      .  - 

s  .  ( 

t     \  ♦ 

.Management  piari  -  Ability  of  the  organization 
to  implement  the  -proposed  program  and  attain 
the  stated  objectives.  » 

•  Does  the  proposers  past  relevant  ex-   .  #  1 
perdence  indicate  a  potential  for  suc- 
cessful* management  and  implementation- 

of  the  proposed  program? 

/ 

+     Dobs  the  propojppr  demonstrate  success,  t 
or  reflect  a  potential  f6r  success  in 
providing  the  described  services  and/or 
training?,  - 

.1  # 

•  Does  the  proposer  demonstrate  success,  * 
or  reflect  a 'potential  for ^success,  in 
serving  the  eligible  population? 

•  Does  the  management  plan  adequately  and 
feasibly  describe  the  necessary  activities, 
events  and  timeline  for  successful  imple- 
mentation of  the  program? 

4  • 

The  foregoing  criteria  Are  provided  only  as  examples. 
Your  PIC  should  develop  its  own  specific  review  criteria  to 
ensure  the  effective  evaluation  of  RFP's. 


Reviewing^  Currant  Deliverers 

The  CETA  Regulation*  require  that  your  PIC  give  special 
consideration  to  those  current  deliverers  or  project  operators 
of  " demon str at eel  §f f ectiveness. "    The  concept  of  demonstrated 
effectiveness  is  left  open  po  local  interpretation. 

Actions  which  PICs1  have  taken  to  give  substance^  tp  the 
conc^Effc.  include:  4 
assessing  compliance  with  CETA  regulations 


/ 


•  ,    evaluating  planned  vs.  actual  performance  - 
v    15%  'deviation  indicating  potential  problems 

•  comparing  the  performance  of  individual 
vendors  in  the  same  category  (e.g.,  class- 
room trainimO. '  1 

One  PIC  established'the  following  guidelines  to  review  \ 
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the  competence  and  projected  ,ef fectiveness  of  existing 

.   .  .  <  ' 

training  deliverers:  ^ 

•  Is  the  tra.ining  .entity  well-established? 

•  Does  the  traininq  entity  have  adequate 
facilities?  \ 

.  •  ,     Will  it  provide  counseling  an£  p]^ce- 
*   ment  services? 

•  Has  the  curriculum  been  developed 
yith  private  teector  input? 

•  Does  the  entity  have  adequate  accounting 
procedures? 

^  •    •       -What  are  criteria  for  instructors? 


wl  atititudinal  and  skill  training  .  * 

be  provided? 

•  Will  training  qualify  CETA  partici- 
pants for  job  openings?-       „  *  <(£ 

•  What  are  *tfre  procedures  fdr  placing 
workers  in  ymsubsidized  jobs? 

An  example) of  a  fo'rm  which  another  PIC  developed  to  review  the 
performance  of  it|»  current  contractors  is  provided  on  the 
followiiyg^page*  £ 

Your  PIC  should  develop  And  utilize  its  own  set  of 
criteria  t<5  sects^  the*best  service  deliverer  possible  from 
among  those  respondents  to  your  RFP. 

Negotiate  and  Develop  Contracts 

After  you  have  identified  the  best  proposer,  you  should 
negotiate  a  contract  or  agreement.    This  process  should 
include  clarifying  any  poijpfs  you  have  regarding  the  RFP  or 
. areas  .in  which  you  want  to  see  the  proposal  modified.  Once 
£his  process  i^s  completed,  you  should  develop  the  formal 
contract.     To  the  extent'  possible,  this  contract  should 
follo.w.the  forrj^t  of  the  RFP  and  inborporate  the  proposer's 
responses  to  the  RFP.     The  contract  should  describe: 

•  Program  description/design  • 
•  •  Goals  and  objectives  *  ^ 
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EXAMPLE 


Projected  Outcomes  fpr*  FY* 81 
Description  of  Performance  hy  Current . Contract  or 


Jurisdiction: 


Program  Activity: 


Planned,  or 
Perf .  Stand. 


A.  Totafl  Enrollments:  « 

1.  Minorities 

2.  Females 

3.  Youth  (21  and  under) 

4.  Older  Workers  (45  and  older)  . 

B.  Total  Terminations: 

1.  Entered  Vnsubsidized  Employment 

2.  Additional  Positive  Terminations 

3.  Other  Terminations 


C.    Entered  Vnsubsidized  Employment 
in  a  Private  Sector  Job 


D.  Expenditures 

^rograr  Activity 

1.  STP 

2.  OJT 


3.  CRT 

4.  EGS 

5.  Total 


Planned  » 

$  , 

$  

s  

$  

$ 


Actual 


$_ 
$_ 
$_ 
$_ 
$ 


.  7 


Actual  Varian 


Variance 


'7 
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•  Outcomes  -  quantitative  performance 
standards,  qualitative  performance' 
standards  dgainst  which  to  Evaluate 
success 

•  How  the  project  will  fit  in  with  overall 
-  PIC  operations  #  . 

•  Target  groups  (if  training  and  develop- 
ment agreement) 

•  Component  design  a^nd  participant  flow 

•  Contractor  responsibilities  -  reporting 
requirements , '  pVogram  <time  f  rames , 
requirements  to  agree  to  monitoring, 
audits,  etc.,  and  CfiTA  Act  and  regu- 

y  lations,  additional  requirements  &nd 
responsibilities  as  necessary  . 

•  Contract^  payment  process. 

•  Participant^|>ayment  process  (if  tfcaining 
aryl  development  agreement), 


CONCLUSION 


Completing  all  the  tasks  within  this  step  in  a  satis- 
factory manner  should  ensure  that  your  PIC  obtains  quality 
assistance  4nd  develops  a  solid  network  of  service  proyiders. 


Employu&enfc  and  .Training  Administration,     / ^ 
CETA  Federal  Representative  Training  Pro- 
gram; Systems  and  Management  Analysis  / 
Skills  Block  (Washington,  D.C. ;  National 
Training  Center,  ETA,  1981) . 

i  % 

Mayor fs  Office  of  Manpower,  City  of  Chicago, 
Planning  and  Evaluation  under  CETA  (Wash- 
ington, D.C:  U.S  Department  of  Labor., 
January  1976) ♦         -  . 
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Finalize  and  Approve  Operations  Plan 


The  purpose  of  this  'step  is  ✓to  develop  a  cotnprehensive 
program  operations  plan  which  can  be  used  to*- direct  all 
PIC  activities  and  services  for  the  fiscal  year.    This  ■ 
step  involves:     developing  work  plans;  developing  final 
cost  estimates,  and  securing  CcAincil  approval  for  the 
entire  operations  plan.  ,  ■  . 


•  PIC  Program  Operations  Plan  -  written  statement  which  docu-* 
ments  PIC's  performance  standards, ,  administrative  structure 
and  organization,  operating  procedures,  .delivery  system,' 

•  work  plans  for  each  project  to  be  undertaken,  management  A 
plan  and  procedures;  evaluation  pl£n  and  procedures;  and 
budge't  allocation  and  accounting  ptocedures. 

•  Project  Work  Plan  -  detailed  statements  of  services/activities 
tftd  be  undertaken,  planned  inputs  and  output^,  work  to  be 

performed  by  individuals,  organization  champs,  work  schedule 
and  action  steps,  administrative  procedures,  and  monitoring 
arrangements/  * 

•  Final  Cost  Estimate  -  review  of  overall  program  costs  prior 
to  approval  of  program  operations  plan.  /J 


J 


Review  the  information . provided  above.  Rate  your  PPC's  knowledge  aftd 
capabilities  With  reference  to  this  step  on  the  scales  below. 


7 


•  Knowledge 


•  Capabilities 


y  1'  

i 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg; 

1 

1  * 

j 

i 

1 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

Based  upon  your  ratings,  decide  -whether  you  need  tc*  read  the  content 
information  on  this-^tep  which  follows  immediately  or  proceed  to  the 
j^ext  step.,  .^f 
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NOTES   (Points'  t&at  are  important  which  you  w^nt  to 

*                                               •  «* 

remember)  ^# 

* 

«  # 

* 

QUESTIONS   (Items  cm  which  you  *are  tuncle^r  or  want 

information)                                           •  , 
* 

9  * 
V 

• 

to  get  additional 

>- 

t 

• 

• 

i 

ACTIONS  (Specific  steps  that  you  intend  to  take  to 
PJC's  PME  process  in  this  area) 

implement  your 

*  • 
'    Steps   .                       Resources/Persons  • 

Time  Frame 

•    >  * 

*»  * 

• 

1  \ 

/ 

t  ' 
♦ 

• 

4 
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,     .  INTRODUCTION  ' 

•  k 

Having  secured  all  necessary  contractor  assistance,  . 
your  last  jftfsffoqisibility  prior  to  actual  program  imple-  ^ 
mentation  is  to  finalize  and  approve  .the  PIC's  program 
operatiohs  plan.    The  primary  tasks  here  are: 

••       to  prepare  work  plans  or  statements 
"  to  guide  all  facets  of  the  PIC's 

^  opetatiort 

•  to  develop  final  detail^^rost  esti- 
mates to  •  ensure  that  you^^lan^is  -  ^ 

within  bjudget. 

After  this  is  accompli sKed,.  the  Cour>cil  should  approve  the 
entire 'operations  plan.  * 

r. 

A.     DEVELOPING  WORK  PLANS 
«  ,  i  • 

Work  plans  are  like  road  maps4    They  provide  a  detailed 
picture  of-  the  work  to  be  done,  procedures  for. doing  it,  and 
preci.se  and  measurable  standards  of  performance  for  each 
activity  which  is' under takfen.     Your  PIC  should  develop  or 
have  work  plans  for  all  projects  and  major  areas  of  activity 
(i.e.,,,  administration,  management  and  evaluation,  partici- 
pant services,  training  and  development  services,  employment 
generating  services) . 

Work  plans  normally  describe: 

•  >    services  to  be  provided/activities 

to  be  accomplished  m 

•  planned  inputs  and  outputs  in  terms 
of  monthly  or  regularltf  scheduled 
objectives 

•  work  to  be  performed  by  individuals 
and  units  *■  responsibility  assignment 

*•       organizational  charts  and  job  descrip- 
tions for  individuals  working  on 
project  or  in  area 
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•  schedule  and  action  steps  for  accom- 
plishing work 

•  administrative  procedures  to  be 
followed 

•  monitoring  arrangements  including 
frequency  and  checkpoints. 

If  your  PIC  follpwed  steps  1-3  in  this  stage,  you 
already  have  developed  or" have  all  the  information  necessary 
to  develop  these;  work  plans.     Your  responsibility  at  this 
point  is.  to  pull  all  of  those  plans  and  data  together  and 
consolidate  them  into  a  master  operations  plan  for  your  PIC; 
This  master  operations  plan  should  also  ^incorporate  the 
management  and  evaluation  procedures  and  checkpoints  from 
your  PlC's  Management  and  Evaluation  Plans  into  standard 
operating  procedures  for  program  operations. 

At  a  minimum,  your  PIC's  final  operation  plan  should  ' 
describe:  -  - 

•  PIC  Program  Performance  Standards 

•  PIC  Administrative  Structure  and  Otganization 

•  PIC  Operating- Procedures  ^ 

•  PIC  Program  Delivery  System  t 

•  PIC  Projects 

•  PIC  Management  Plan  and  Procedures 

•  PIC  Evaluation  Plarwand  Procedures  ^ 

•  Budget  Allocation  and  Accounting  Procedures 


5ost^ 


B.     FINALIZING  COST  ESTIMATES  AND  COST  PLAN 

.  / 

As  part  of  putting  together  your  PIC's  operational  plan, 
you  need  to  do  one/more  check  on  costs  to  be  certain  they  are 
within  the  permissible  and  planned  ranges.     To  accomplish  this, 
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you  should;  ,  * 

•  document  all  final  project  costs  as 
dettfAined  through  contract  negotiation 

•  and  ccxoj^are  them  to  your  original  esti- 
mates zo  >not£  any  significant  differ- 
ences 

*  » 

•  display  all  of  the  line-item  costs  of 
*       the  program  to  ensure  that  funds  are 

being  planned  to  generate  services  at  ^ 
maximum  levels  of  efficiency  and  «Jf 
effectiveness:  * 

.-  Administration  * 

-  Allowances 

-  Wages 

-  Fringe  Benefits 

-  Training  ft 

'  -  Services  A 

•  array  costs  within  the  proper  categories 
to  ensure  that  all  projected  costs  are 
in  accordance  with  financial  provisions 
of  the  Fjederal  Regulations: 

Participant-Related  -  at  least  50% 
-v_    EGS  -  up  to  30%  , 

Administrative  -  no  more  than  20% 

*  •        ensure^ that  your  total"  cdst  does  not  ex- 

ceed your  total  Title  VII  grant  alloca- 
tion, unless  other  available  resource's 
(federal,  state,  or  local  -  public  or 
private)  have  been  committed  to  the 
progfam.  v 

If  your  analysis  reveals  any  problems  in  the  budget, 
you  should  make  the  last, minute  adjustments  that  are  neces- 
sary to  bring  your  total  program  into  compliance. 


C;     APPROVING  THE  PLAN 

'  « 

Once  the  operations  plan  is  developed  and  the  final 
costs  are  determined,  the  entire  package  should  be  submitted 
to  the  Council  for  approval.     If*  the  Council  has  participated 
all  along  in  the  drafting  of  the  plan,  t£en  this  step  is  <  — * 
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primarily  symbolic.    The  formal  seal  of* approval  for  the  PIC's 
operations  plan  by  the  Council  indicates  that  all  membe.rs  endorse 
it  as  a  plan  and  program  approach  which  they  can  support. 

-  Again,  as  with  nearly  all  of  the  earli'er  steps  and  activi- 
ties, the  actual  process  for  securing  this  approval  should  be 
determined  by  what  you  anticipate  will  work  best  for  you*  A 
few  suggestions  regarding  this  approval  process  follow. 

•  Involve  the  Council  or  subcommittees  at  * 
all  important  points  throughout  the 

process  of  plan  development,  raising 
^         key  issues  for  their  resolution. 

•  Ensure  that  the  members  get* copies  of 
the  draft  plan,  in  advance  of  the,  /  * 

approval  meeting.  ,  * 

Have  the  plan  approved  in  a  full  Coun- 
cil meeting. 

After  the  meeting,  disseminate*  the 
approved  plan  to  all  members  and  to 
public  officials  and  others  who  you 
#feel  should^ be  informed*  of  the  PIC's  \ 
activities. ?  **  * 


CONCLUSION 

# 

A  final  note  on  your  program  operations  plan  is  that 
although  considerable  time  and  effort  go  into  creating  a  plan  m 
which  is  as  appropriate  and  responsive  as  possible  to  local 
needs,  the  plan  is  based  upon  factors,  assumptions,  and  informa- 
tion at  a  given  point  in  time.   *The  world  is  not  static.  Changes 
in  the  economy,  size  and  needs  of  target  groups,  and  CETA  funding 
levels,  among  other  things,  will  cause  the  ieed  for  replanning. 
Therefore,  you  should  be  prepared  to  replan.    When  the  need 
arises,  your  PIC  should  do  its  replanning  using  the  same  process 
in  miniature  that  you  employed  to  develop  its  original  program 
operations  plan* 
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# Employment  and  Training  Administration, 
v CETA  Federal.  Representative  Training  Program; 
Grants  Management'  Administrative  B^ock         •  ' ' 
(Washington  *^D. C. :  Natianaj.  Training  Center, 
*   .ETA,  1981)>T 


Manpower  Administration  (now  Employment 
ah,d  Training  Administration  [ETA]), 
Manpower  Program  Planning  Guide  (Washing- 
ton, D.C*:  U.S.  Department  of  LabQr^ 
April,,  1974)\  ^ 


Mayor's  Office  of  Manpower,  tJitJ  bf*Chica^6,  Planning  and 
Evaluation  under  CETA  <Washington,-*B.*C  :*  U.S.  Department  of 
Labor / 'January  1976).  -  * 
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E.     DEVELOP  MANAGEMENT  PLAN 


Purpose:  ^  To  ^erbetMAgh  a  detailecT* plan  for  overseeing 
it  PIC's  program  operations  on  an  ongoing^ 
isis. 


.1.    Establish  Management  Review  and 
*  Control  Process 

2.  Determine  Management  Information- 
and  Reporting -Needs 

3.  Design  Management  Reporting  System 
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Step  1 


Establish  Management  Review  and  Cofttrol  Process 


The  purpose  of  this*  step  is  to  establish  an  operational  con- 
trol process  and  steps  for  Aanagement  review  of 'your  PIC's 
program.    This  involves:     identifying  the  basic  procedures  * 
required  to  monitor  and  control  program  performance;  and 
dividing  responsibilities  for  management  review  and  control 
among  PIC  members  and  staff. 


L 


•  Operational  . Control  Process  *-  method  for  assessing,  directing 
and  correcting  PIC  program  operation  on  an  ongoing  basis. 

•  Management'  Review  Responsibilities  -  roles  and  tasks  to  be 
performed  by  PIC  staff  and  members  in  overseeing  and 
coordinating  program  operation  elan. 

•  Report  Monitoring review  of  planned  versus  actual  performance 
using  reportfe  submitted  by  projects  and  contractors. 

•  Preventive  Monitoring  -  regularly  scheduled  visits  to  projects/ 
contractors  to  oversee  and  review  interim  products  and  work 
processes. 

•  Corrective  Monitoring  -  follow-up  analysis  to  determine  why 
performance  varied  significantly  from. planned  levels.       *  \ 


f 

C(.  U   Km>wlVdge/Capabil1t1es  Rating  ^_^> 

Reviet^the  .information  provided 
capabilities  with  reference  to 

above.     Rate  your  PIC's  knowledge  and 
this  step  on  the  scales  below.  # 

•  Knowledge                    |  . 

1 

*  « 

i  1 

Excel- 
lent 

Above  , 
Avg*. 

Aver- 
age 

Below 
Avg. 

Poor 

•  Capabilities  4 

I 

1 

» 

 1 

Excel- 
lent 

Above 
Avg. 

Aver- 
age 

Below 
Avg. 

Poc^ 

Based*  upon  your  ratings,  decid^ 
information  on  this  step  which 
next  step. 

whether 
follows  * 

you  need  tb  redd  the  content 
immediately  or  proceed  to  the 

ERIC 
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NOTES  (Points  th^t  are  important  which  you  want  to  remember) 


Q^ESTION^^Ttem^o*^^ 

~~    information)  ,  f 


...  •  < 


* 

r 


/  i 

r 
i 


ACTION^^SpecSi^^tep^ 

PIC's  PME  process  <oJ»t§hi^  area) 


Steps 


-Risoi 


sources/Persons 


Time  Frame 


<  * 
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INTRODUCTION 

There  are  two  fundamental  questions  which  your  PIC  has 

to  ^nswer  to  establish  its  manageiflent  approalch: 

/  ' 

•  What  will  your  PlCfs  basic  operational 
control  process  and  procedures  be? 

•  Who  will  be  responsible  for\  implementing 
■  and'  administering  that  process  and  what 

will  their  specific  responsibilities  te? 

Your  answers  to  these  questions  will  shape  your  PIC's  general 
framework  for  managerial  oversight  and  programmatic  control. 


A.     OPERATIONAL  CONTROL  PROCESS  -  OPTIONS 

An  operational  control  process  is  a  method  for  assessing, 
directing,  and  correcting  progr^a  operations  on  an  ongoing 
basis. *  As  discussed  earlier,  regular  monitoring  of  perform- 
ance versus  plan  should  be  the  core  of  your  PIC's  operational 
control  process.     In  addition  Vo  that  paper  technique, 
"depending  upon  the  resource*  which  you  have  available  and 
the  control  needs  of,  your  PIC  as  you  define  them,  .ypu  will 
want  to  engage  in: 


preventive  monitoring 
corrective  monitoring. 


Performance  versus  Plan  -  Report  Monitoring 

In  performance  versus  plan  monitoring,  ^ou  compare  your 
PIC's  performance  against  the,  quantitative  and  qualitative 
targets  expressed  in  your  program  operations  plan.     If  you 
have  followed  the  PME  Process  to  this  point,  your  PIC's 
program  operations  plan  contains  near^-terra  targets  for  both 
your  Participant  and  your  EGS  projects. 

v'  •' 
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To  implement  a  simple  petf ormance  versus  i^an  monitoring 
system,  you  t£§ck  your  A'IC's  performance  against  those  objec- 
tives which  you  have  sjt.     This  a$n  be  done  by  reviewing  reports 
which  are  generated  to' meet  Department  of  Labor  reporting  re- 
quirements and  other  special  -reports  which  you  can  require  of^  ^ 
program  operators.  -  ' 

This  form  of  analysis  wiil  enable  your  PIC, to  identify  those 
operational  areas  in  which  there  may  be  performance  problems. 
However,  this  examination  of  performance  versus  plan  as 
reported  oi>  paper  will  not  enable  you  to: 

•  prevent  problems  before *they  occur 

•  identify  the  specific  reasons  or  causes 
for  performance  below  plan. 

^complish  these  ends  satisfactorily,  your  PIC  i»uld  have  to 

e^and  its  managerial  monitoring  capabilities. 


To  crqcc 


Preventive  Monitoring  ***** 

Preventive  monitoring  is  a  mfethod  for  directly  overseeing 
the  efforts  of  contractors  through  on-si^e  visits,  or  scheduled 
meetings  to  review  interim  products  and  work'  processes.  This 
method  is  applicable  to  both  your  PIC's  EGS  and  Participant, 
projects.     It  jean  be  used  to  pinpoint  and  eliminate  short-term 
problems  and  to  redirect  prog^m  efforts. 

To  facilitate  its  preventive  monitoring,  your .PIC  can  create 
personnel  eva"iuation  and  monitoring  checklists  wW^ch  can  be  used 
during  the  monitoring  visit.     These  checklists  could  be  designed 
to  direct  your  attention  to  those  areas,  of  activity  which  are 
most  important  -to  ensure  successful  performance  of  staff  and 
project  operators,  f  # 

Corrective  Monitcj^ing  0 

The  purpose  of  corrective  monitoring  is  to  determine  why 
performance  varices  from  planned  levels  and  to  identify  possible 
remedial  actions  for  management  consideration.     Although  there 
is  no  set  process  for  corrective  monitoring,  steps  which  your 
PIC  might  take  in  doing  corrective  taonitorirfg  include: 

1.     Review  Performance  versus *Plan.  Report 

from  project  operators ;^  ^  -  ' 
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2.  Note  significant  variation  from  plan. 

3.  Talk  to  project  operators  to  ascertain 
possible  reasons  for  variance  and  to 
set  up  a  meeting. 

4.  Meet  to  discuss  possible  causfes  and 
solutions.  , 

.  1  '  : 


5.     Resolve  the  issue. 


^6.    Agre^upon  informal 
corrective  action.- 


Initiate  further  in- 
vestigation. 

Conduct  personal  inter- 
views and  data  review. 

Draw  tentative  qonclu- 
sions/solutiofts . 

Discuss  vy.th<  contractor . 

Develop  corrective  ac- 
tion report.  • 


B.     DEFINING  YOUR  PIC'S  OPERATIONAL  CONTROL  PROCESS 

To  review,  your  PIC's  operational  control  process  can 
consist  of  three  basic  cpmponents,  as  shown  in  the  .diagram 


below, 
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You  should  decide  which  components  should  be  in  your  PIC's 
process  based  upon  local  needs  and  capabilities.  ■  At  a  minimum, 
you  will  want  to  be  able  to  do  some  performance  versus  plan 
and  corrective  monitoring.    After  you  have  defined  the  types  of 
monitoring  your  PIC  will  do  ' —  the  process  to  be  employed  — 
you  should  spell  out  the  basic  procedures  which  will  be  followed 
for  monitoring  purposes.     To  do  this,  you  should  answer  the 
following  questions: 

What  information  is  absolutely  essential 
for  monitoring  performance  and  managing  * 
the  PIC's  program?  >  * 

What  are  the  sources  o&  this  information? 

How  can  this  information  be  collected? 

What  procedures  are  necessary *fb£  analyzing 
the  information?  4 

When,  how  often,  and  in  what  irtanner  should 
the  information  be  reported? 

Who  will  review  management  reports?    In  par- 
ticular, what  role  will'  the  Council  members 
\  Rlay  in  the  process? 

\  How  will  corrective  actions  be  implemented? 

After  you  hav€?  answered  these  questions,,  you  should  prepare  a 
written  description  of  operational  control  procedures  for 

,  J 

your  PIC.      ,  •  * 


C.     ASSIGNING  MANAGEMENT  REVIEW  RESPONSIBILITIES 

After  you  have  made  these  -decisions,  you  will  have  the 
information  necess^y  to ,  determine  basic  mariagement  review 
responsibilities  for  both  PIC  staff  and. members .  Questions 
you  will  need  to  answer  in  this  area  include: 

rm  Who  will  have  ultimate  responsibility 
for  administering  the  monitoring  plan?  1 

•  Who  will  supervise  monitoring? 

^•'Who  will" prepare  monitoring  and  manage- 
ment reports? 
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*  Who  will  be  involved  in  the  analysis 
of  monitoring  reports-  anjl  in  the 

/  interpretation  of  findings? 

•  What  will  the  Council's  role  be  in 

the  monitoring  process?  ,  m 

\  Obviously,  how  you  break  out  the  management  responsi- 
bilities for  implementing  your PIC s  operational  control 
procjess  will  be  based  upon  a  number  of  factor's,  including: 

•  the  nature  of  th6  control  process 

•  your  PIC's  size 

•  available  resources 

•  •     staff  capabilities. 

One  possible  ro\e  and  Council  arrangement"  for  accomplishing  1 
the  management/dontrol  function  and  basic  considerations 
related  to  using  PIC  staff  for  monitoring/control  purposes  . 
are, outlined  on  the  following  pages. 

Council  Role  and  Responsibilities 

Regardless  of  your  PIC's'  control  process,  you  should 
ensure  that  the  Council  plays  a  central  role  in  the  regular 
monitoring  of  program  performance.     In  this  regard,  it  jnay  * 
be  advisable  to  have  a  council  subcommittee  charged  with  over 
seeing  program  performance.     Responsibilities  of  such  a 
committee  could  include:  ^ 

•  Determining  an  operational  control  process 
for  FlC  to  be  recommended  to  the  full  Council. 

•  Reviewing  monitoring  designs  developed 

By  staff/^tad  to  advise  the  Council  with  J 
regard  to  these  designs.* 

•  Making  recommendations  to  the  Council 
with  regard  to  the  selection  of  eval- 
uation criteria ^nd.  performance  measures. 

•  Receiving  and  review  regular  management 
reports  on  program  performance  to  assure 
its  adhereixre  to  the  plan. 

^      Reviewing  corrective  action  reports  ahd  '  . 

to  dall  the»  to  the  attention  of  the  - 
*  full  council  as  appropriate. 
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Staff  Role  and  Responsibilities 

Assigning  staff  management  review  and  control  responsi- 
bilities is  .possibly  the  most  critical  part  of  establishing  * 

* 

your  PIC's  management  review  and  Control  system.  Staff  make 
your  PIC' s  operational  control  and  management  review  process 
work.*    This  assignment  is  a  three-level  process:  w 


1.  Determine  the  monitoring  and  control 
tasks. that  must  be  performed. 

2.  Identify/select^ staff  tftat'are  capable 

of  performing  these  tasks . 

*  f 

3.  Allocate  and  make  staff  accountable 
for  work  performance  through  job 
descriptions  and  performance  planning. 

'      1 .     Monitoring,,  anci  ContrQl  Tasks 

In  general,  tfhe  tasks  wljioh  PIC  staff  will  be  required  to 
perform  in  the  monitoring  and  control  area  are  to: 

•  design  njonitbring  and  control  system  and 
manage  its  implementation 

prepare  management  reports  ^ 

•  analyze  report  data* and  translate  that 
•data  into  conclusions  and  recommendations 

•  develop,  as  required,  corrective  action 
plans  for  remedying  problem  situations 

•  conduct  on-site  and/or  personal  visits. 
^  and  interviews  to  obtain  monitoring  and 

assessment  data 

•  design  dat:a  collection  instruments  and 
report  formats  -  *  ^  ^ 

•  collect  management  and  monitoring  data  . 

\ 


J 
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*2.     Staff  Capabilities  ™ 

The  general  capabilities  required  of"  PIC  staff  to  perform  * 
these  tasks  adequately  include  knowledge  and  abilities*  re- 
lated to:  * 

•  CETA  Law,  regulations,  .planning  and 
reporting  requirements 

•  principles  and  elements'  of  manpower 
program  design 

•  basic  quantitative/qualitative  statis- 
tics and  analytical  techniques 

•  performance  objectives  and  indicators 
0         performance  assessment  methods 

•  monitoring  procedures 


3 .    Work  Distribution 

fcThe  actual  distribution  of  monitoring  work  to  staff 
will  depend  upon  the  size  and  configuration  of  your  PIC, 
Once  you  have  identified  those  tasks  which  have^  to  be  per- 
formed, and  identified  the  staff  .to  perform  the  tasks, 
you  should  allocate  work  accordingly. 

Two  vehicles  which  you  can  use  to  help  facilitate 
PIC  staff  performance  in  the  monitoring  area  are: 

•  job  descriptions  ^ 
<  •       performance  planning." 

Job  Descriptions:     One  of  the  best  ways  of  helping  indi- 
vidual PIC.  staff  to  understand  and  perform  their  management 
review  responsibilities  is  to  include  these  responsibilities 
in  their  job  descriptions.     At  a  minimum,  the  30b  description 
should  include:  # 

•  a  complete  list  of  the  relevant  manage- 
.  *  "  ment  review  responsibilities;  and 

•  standards  for  the  performance  of  these 
responsibilities. 


Ill 
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Performance  Panning:     Job  descriptions  provide  the  written 
record  of  assigned  PIC  monitoring  responsibilities.     However,  # 
in  order  to  translate  that  written  record  into  job  action,  it 
is  alsa  necessary  to  plan  this  work.     "Performance  Planning" 
i's  a  process  which  you  can  use  to  clarify  job  expectations  an£ 
standards  w£th  staff  and  plan  for  their  accomplishment.  Essen- 
tial steps  in  performance  planning  include: 
•  • 

a.     Clarify  job  objectives  and  standards  r 
explicitly  define  those  monitoring  and 
control  tasks  that  the  staff  person  is 
^fco  perform  as  well  as  how  well,  those 
tasks  are  tcf  be  performed; 


b. 


c . 


Establish  action  steps  and  timetable  - 
mutually  work  out  monitoring  plans* and 
timetables;  •  ,  '  ^ 

Efaphasize  accountability  -  make  it  clear 
that  the, staff  person  is  responsible  and 
accountable  fori  accomplishing  monitoring 
^nd  cantrol  tasks  as  they  were  discussed. 


CONCLUSION 

tJj\  conclusion,  once  you  have  established  the  operational 
control  process,  and  divided  responsibilities  for  management 
review  among  your  PIC  staff  *and  members,  you  have  developed 
the  infrastructure  necessary  for  an  effective  program  manage- 
ment system. 


Manpower  Administration  (n6w  Employment  and 
Training  Administration   [ETA] ) , . Program 
Assessment  Guide  (Washington,  D.C. :  U.S. 
Department  of  Labor,  April,  1974). 

Olympus  Research,  Self -Evaluation  of  CETA 
Manpower  programs;     A  Guide  for  Prime 
Sponsors  (Springfield,  VA:  U.S.  Department 
of  Commerce,  1975) . ■ 
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Step  2 


Determine  Management  Information^and  Monitoring  Needs 

The  purpose  of  this  step  is  toilet  ermin*  exactly  what  informa- 
tion your  PlC  needs  to^ ensure  effective  Hfigoing  program 
ifianagement  and  accountability.  "  In  this  step,  you  review 
program  and  project  objectives,  plansy^-and  performance 
standards  in  order  to  select  indicators  or  assessment  areas 
in  which  measurements  are  to  be"  made  on  a  regular  basis  for 
monitoring  purposes. 


Management  Information  and  Monitoring  Needs  -  minimum  amount 
of  data  needed  to  review  PIC  programmatic  effectiveness  and 
to  exercise  control  over  program. 

Program-Level  Monitoring  -  review  of  overall  performance 
of  all  facets  and  projects  constituting  the  PIC's  programs 
for  the  fiscal  year.  >\ 

Component- l£vel  .Monitoring  -  revie*W  of  major  facet  (e.g., 
classroom  training,  linkage  projects)  of  PIC  operations. - 

Project-Level  Monitoring  -  review  of  performance -of  indi- 
vidual contractor  or  service  deliverer. 


M(  fl  Knfewledgt/CapabHUIes  Rating 

Rqview  the  information  provided  above.  Rate  your  PIC's  knowledge  and 
capabilities  witji  reference  to  this  step  on  the  scales  below. 

•  Knowledge 


•  Capabilities 


i 

i 

1 

»  i 

1 

Excel- 

Above 

Aver- 

Below 

Poor 

lent  / 

Avg. 

age 

Avg-. 

1 

I 

 -J  

1 

 J 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

Based  upon  your  ratings,  decide  whether  you  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the 
next  step. 
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*                PME  PROCESS  NOTES/ ACTION  rJUANil  INv*  r/\ijr*  % 

1  r 
1  f 

NOTES  (Points  that  are  important  which cyou  want^to  remember) 

• 

< 

• 

r  1 

*  - 

QUESTIONS  (Items  on  which  you  are  unclear  or  want  to  get  Additional 

information)  ^ 

* 

* 

* 

* 

/                                                      :  ' 

ACTIONS   (Specific  .steps  that  you  intend  to  take  to  implement  your 

PIC's  PME  process  in  this  area) 
'si-^p*                          Resources/Persons                   Time  Frame 

• 

* 

*  * 

*  • 

*  * 

• 
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INTRODUCTION       *  <J 


•Since; it. is  impossible  to  monitor  totally  all  activities, 
components ,  and  projects  of  the  PSC,  your  primary  responsibility 
in  this  step  is  to  define  the  Jainimum  information  that  -you 
will  iequir^  t^'te^L  how  well  your  PIC  is  doing  in  order  to^ 
be  able\to  exercise  effective  control  over  prpgram  per- 
" f online e.  ^  ^his  section^begins  with  an  oversew  of  the  "steps 
that,  yBymy^^lia^ake  to  define  thRse  management  information 
and  monitoring "J!needs  and  then  furnishes  specific  infori^tion 
.'^pn  how  yo\^f  PIC  can  go  'about  decidiiflj  what  to  monitor.  j 


*    9    a.     DEFINING  MANAGEMENT  INFORMATION  AflD     0  ' 
'     NEEDS  -  THE  PROCESS 

/The  steps  involved  in  defining  your  PIC  *  s  "basic  'manage- 
ment  information  and  control  -needs  are: 


\ 


-  %  identifying  the  data  and  information  - €hat  . 
you  need  in  order  to  adequately  control 
1         the  program;  and       *  ,      '  - 

•  determining  where  this  date  and  informa- 
tion canv  )ae  found  and  how  to  obtain  j%. 
»  •  jr  * 

1  3  m  V  * 

*  Identifying  Datq  Needs    m  * 

The  initial  phaie  of  identifying  your  c(ata  needs  Ts— 
^asking  yourself ^the  following  questions:    Whaj^do  you  need" 
to  know?    What  quest'^is  will  the  data  be  expected  to  ariswer? 
'Once  you.  have  specified* the  answers  to  these  questions,  you 
have,  in  fact,  determined  your  data  need's. 
V*  '    :*   In  general,  throne  question  that  all  monitoring  and 
control  djata  Aauffiswer  is:  -  How  close  are  you  to  meting 
your  program  goetrs?    What  this  means  is  that,  at  a  minimum, 


0 
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you  should  collect'  data  on  the  participant  side,fcf  your  pro- 
gram  which  tells  you;  Jj*  t 

•  Who'  hSs****©  served?     (participant  characteristics) 

•  How  mart^have  been  served?     (enrollment  data) 

•  'What  h^Mbegn  the  pi*ogramvoutcom^s?.  (deter- 
minatWn  .and  completion  data) 


•  What  ha§  it  cost?     (expenditure  data) 

•On  the  employment  generating  services  side,  you  will 
want  tQ  know:  . 

•*         •  WKa^  products/activities  have  been  completed? 

•  What  have  been  the  results? 

•.Were  they. completed  in  a' timely  and  high 
quality  manner? 

•  What  has  it  co§t?  J 


Identifying  Data  Source's   *  .  f     *  *%  4 

W  9  '  Once  you  have  determined  the  information  you  will /need. 

— -V~ift  order  to -monitor  your  program,  ^ou  will  need  to  identify- 

where  the  data  can  \^e  fdund.     In  identifying  youp  data  sources,  * 

you  will  aLso  be  determining:  2  %  Q  » 

*  •       whether  the  data  can  be  generated  by  current 

reports  required  by  the  Department  of « Labor;  or  *  '  ,  « 

•  v      if  new  reports  must'be  designe^to  dollect  the 
data  you  need.  ^ 

*  v  -  .  v        ,  - 

•  Aihong  the  major  sources  of/An  format  ron  on  the  planned*  afi<3  actual  ♦ 


per  formate  of  your  p^ografm  are: 

•  participant  records  • 

•  agplicant^records  . 

•  MlS-generated  teports  on  participant  tracking 

•  contractor  generated  reports  on  financial. 

expend  i  ture  s 

,  •       progress  reports  from  contractor  or  staff  * 
on  EGS  projects 
on-site 'observations  and  monitoring  visits 

•  intferviews  with  applicants,  participants,  staff 
interviews  with  employers. 
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-For  the  most  part,  the  data* that  you  choose  to' collect 
and  what  you  will^  monitor  will  be  dictated  by  the  performance 
objectives  and  standards  contained  in  your  PIC's  program 
.operation  plan  for  the  fiscal  year. 


B.     DECIDING  WHAT  TO  MONITOR  AND  HOW 


As  stated  at  the  outset  of  this. section/  yotlr  PIC's 
primary  form  of  monitoring  will  be  comparing  actual  against, 
planned^performance  to  determine "the  degree  to  which  standards' 
or  oBjfectives  are  being  nteti     Regardless  of  the  exact  content 
Of  y<?ur  PIC'-s  program  operation  pl^h,  you  will  want  to  monitor 
performance  at  three  le^Jls:  r  ~ 

•  pgflggam  (overall) 

•  component  (major  facet,  e.g.  r  classr'oom  training) 
*    •         service  deliverer/project . 

Therefore,  the  first  step' to  take  to  identify  you^  PIC^s 
specif ic"  management  information  and  monitoring  needs  is  to 
review  your  PIC's  program  operations  plan  to  determine  pre- 
cisely  what,  at 'each"  level ,  ought  to  be  measured 'and  reported. 

Not  all  areas  can  or  should  be  monitored.     The. most 
efficient  metho<J  for  defining  Reporting  and  monitoring  needs 
is  to  assess  each  area  against  the  following  criteria v 

•  What  areas  must  be  measured  and  reported  r 
by  law?  ; 

What  areas,  if  measured,  wij.1  give  you 
v         reliable  information  on  how  the  program 
,     *         is  progressing?  . 

•  What  areas  should  be  monitored  because 
they -are 

—  new, 

—  experimental, 

•  —  politically  sensitive,  or 
— >  expensive* in  terms  pf  resource  investment? 
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Using  these  criteria  will  fielp  you  to  determioe^hat  to  monitor. 
Once  you've  identified  the  areas  at  each  of  these  levels 
w in  which  you  need  to  jaonitor,  the  next  step  is  to  define  for 
each  area  to  be  measured  such  as:  * 


objectives/activities  to  ,be  accomplished 

•indicators  of  ^perf  ormance 

phe  information  you  will  need  in  order  to 
monitor 

•  the  source  iot  this  information  '(how  tA) 
'    collect  it)  and  the  frequency,  with  which 

you  should  collect  it 

•  a  checklist        information  availability. 

The  discussion  which  follows  descrribes  methods  and 
measures  that  your  PIC  might  employ  to  establish  its  monitoring 
.procedures  at^all  three  levels. 


Program  Level  Monitoring  -  Overall  Ferformance 

Deciding  what  and  how  tp  monitor  at  the  program  levej. 
will  be  the;  easiest  determination  for  your  PIC.     If  you  set 
program  performance  jstandards  at- the  outset  of  -the  fiscal- year,  < 
thea  they 'are  what  you  want  to  mohitor  against  on  an  ongoihg 
basis.  .  < 

To  repeat,  performance  standards  specify  end  results  ex- 
pected and  can  be  se£  foi?  items  such  ass 

enrollments  levels  »  "  ' 

expenditure 'levels         *        %  *  f 

levels  tof  service  to  target  groups     ^  t 
positive  termination  rates*'       "  / 
•  .  entered^  employment  rates 

'  cost  per  participant  „  .  . 
cost  per  positive  termination 
response  r$te  to  local  employer  survey  v 
number  of  jobs  added  throjigh  economic  developmer>t 
number  of'  informal  or  formal  planning  "kgreemertts 
established. 
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To  monitor  against  those  items,  you  measure  ptogress  at 
regular  intervals.    Much  of  the  data  that  you  wilL  need  to 
monitor  in e these  areas — especially  on  the  participant  side — 
can  be  easily  retrieved  through  the  reports  prepared  on  a 
monthly  and  quarterly  basis  for  the  Department  of  Lkbor. 
Other  data  can  .be  secured  through  special  reports  which  can 
be  required  from  contractors  or  staff. 


Component/Project-Level  Monitoring  \ 

Deciding  what  and  how  to,  monitor  at  the [component  and 
project  level  is  a  somewhat  more  complex  process.     Again,  . 
however,  if  you  have  developed  your  PlC'Jb  program  operations 
plan  according  to  the  procedures  outlined  in  this  Guide,  then 
there  is  a  straightforward  method  which  can  be  used  to  reach 
these  decisions  .# 

Your  Program  Operations  Plan  will  consist  of  complete 
work,  statements* for  all  major  areas  of  activity  and  projects. 
These  work' statements  will  spell  out: 

•  "services  to  be  provided/activities  to  be  . 
accomplished 

*  .A 

„    m       objectives  to  be  echieved/perf  quittance 
*  standards 

•  '     planned  inputs  and  outputs  in  terms  of 

.         i  monthly  or  regularly  scheduled  objegi-  ^ 

'  tives 
» 

.•  work  to  be  performed  by  individuals 

„   '    _  and  units 

\       .  #        schedule  and  action  stkps  for  accom- 

plishing work. 

'    *  You  can  use  the^work  statements  for  those  compoh^nts 

and  projects  which  you  have/decided  to  monitor  to  help  ff&<ine  - 
your  management  information  and  reposing  methods.  Specifically, 
by  comple^^F  a  'worksheet  ^milar  to  that  on  the  following 
page  for  e^ch  area  to  bB  monitored,  you  can  establish  your 
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PIC's  component  and  project-level  management  informatiqn  and 
monitoring  f ramewox^e.  *  ' 

To  complete  the  worksheet  foiTeach  component/project,  you 
would  take  the  following  steps;  *  ♦ 

*     1.     Read  the  work  statement  to  identify  all 

major  purposes  or  activities.     Enter  them 
in  column  1.  ... 


2. 

„/ 
3. 


5. 


^Identify  dbjectives/perfdrraance  standards 
for  each  ' activity. -  Enter  tjiem  in,  coliimo  2. 

Determine  what  measures  or  indicators  can 

used  to  assess  achievement  of  objec-  '  " 
tives.     Enter  in  column  3-. 

Decide  what  data  or  information  will  be 
needed  to,  providg  Evidence  £hat  the    .„„   *  , 
indicators  or  standards  have^Jbaan  met. 
Indicate  that  in  colucm  4. 

Determine  where  data  can  be.  secured,  how 
it  srtiould.be  secured,  how  oft£n  it  should 
be  collected.     Enter  that->  in  column  5. 

Ascertain  whether  tnformation  is  currently 
reported,  or  whether,  you  wij.1  have  to 
devise  special  report  format-.     Check  appro- 
priate box  in  column  6.  • 
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An  example  of  a  completed  worksheet  follows.    Once  you 
have  completed  worksheets  for  all  components/projects,  then 
you  are  ready  to  design  y,our  PTC's  overall  management  reporting 
.system  and  monitoring  plan. 


to 


■  > 
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<TaV* 

COMPONENT  PERFORMANCE  OBJECTIVES  /  INDICATORS  WORKSHEET  ^  _f&b'J, 


To  pro.ide  *lMiM*»t 
unices  and  assign  *»« 
r*rer  111  participants  to 
at«s«r  coeoonents  m 
*  accordance  ,nd1v1" 
^lizA  service  strategy 
developed  fry  tne  [EOP] 
counselor  with  the 

%   De*e1ot  detailed  com- 
ponent Jesign  W4t  pro- 
vides  md*Ptn  orientation 

8    Oe.elOP  en  COP  ooc-«* 


>*tiop  : 
•  i  CA  all  c 


cations 
MS 


4rvn<it  and  tram  stef'  v 


Provide  to  de- 
tailed ortentltlon 
program  designs  and 
curricula  for 
orientation  program 
operate  program  to 
serve  4.020  partlct 
pants  per  year 


1  8 


Develop  In  EE*  docu- 
ment meeting  speci- 
fications outlined 
in  CETA  Kf  P  end  '  ^ 
Component  Wort  State- 
ment for  COP  com- 
ponent * 


Provide  nonthly 
status  reports  to 
all  components  re: 
participants  ready 
for  referral  to  the 
coaponent.  pro* lens 
with  rpferra) 
schedules,  etc 


Orientation  program 
designs  and  curricula 


•  Activity  eecords  for 
•r*  en  tat  ion  program 
aftd  curricula 

•.Contract  with  COP  Com- 
ponent including  pertly 
clpent  loafing  plans 

•  Observation  of  orienta- 
tion prograa 


Usable  COP  document 
eveting  ell  specifica- 
tions in  RF»  and  work 
statement 


Monthly  written  s tatu* 
reports,  evidence  of 
regular  effective 
communication  with  otmer 
component  staff 


1  0 


Trim  all  COP  staff 
m  participant 
assignment  standard 
for  eacn  coaponent 
and  to  administer 
appropriate  testing 
and  assessment  to 
determine  partici- 
pant's *>eeds 


All  participants 
referred  to  com- 
ponents of  the 
PIC" s  delivery 
systems  must 
meet  the  speci' 
fied  participant 
assignment 
standards . 


a  Copy  of  tDP  document  in 
use 

Copies  fit  WP  and  wort 
statement  specifications 
for  COP.  document 


a  Copies  of  written  en 
munication 

a  Verification  of  comm 
cation  from  other 
component  staff 


C    Determine  Supportive 
services  neeos  of 
participants  and.  «*ke 
appropriate  ^eterrals 


«<-er  parti  ;ipVn; 
otr.ep  co«5o«ents 

KC^fling 

Dasec  pn  ■* 
standards 


s  to 
I J  and 


Develop  list  of  local 
supportive  service 
providers,  enter 
'nto  agreements  for 
referral  of  C£TA- 
participants 


Refer  all  partici- 
pants to  appropriate 
components  with  Jess 
•nan  15  percent 
error  rate  \ 


J? 


a  Assessment  and  character 
istics  dau  from  COP  and 
application  form  on 
par;lc1pants  rtftrrt^ 
from-CDP  to  other  com- 
ponents, form  Showing 
component  tb  which 

a  Roster  of  participants 
rtftff^  during  the 
monitoring  period 


Assignment  standards 
from  wort  statements  for 
eacn  c 


a  Quarterly  desk  review  of 
attendance  records  for 
orientation  and  component 
activity  Summaries 

a  Quarterly  onsite  monitor- 
ing of  orientation,  pro* 
9 ram  to  ensure  con- 
formity with  curricula 


f  Annual  desk  review  of 
document* 

a  Quarterly  onsite  refieV 
to  assure  its  current 
use 


•  Quarterly  onsite  reviews 
of  written  communication 
from  COP 

a  Ouarterl/  interviews 
with  sample  of  0 
staff 


a  honinly.  onsite  review 
of  s«»*i1<  of  2  percent 

or  mare  Of  records  of 
participants  rtftrrtC 
during  monitoring  period 

a  Roster  from 
-^irtiich  to  draw 
sample 


/ 


/ 

/! 


/ 


A 


■r  Evidence  of  linkages 
between  CDP  component 
and  local  supportive 
sendee  providers 

a  Documentation  of 
assessment  and 
referrals 


•  No  more  tnan  *  percent 
of  tnose  enrol  led  in 
any  component  will  he 
referred  back  to  COP 
within  the  first 
three  days 

•  Ho  more  than  15  per-* 
cent  of  those  enrolled 
in  any  component  wi 1 1 
he  -efoj-red  hack  during 

•  the  duration  of 
component  service 


a  Assessment  data  from 

CQP's  and  applications 

a  Documentation  oi 
referrals  to  supportive 
services 

a  1,1st  of  providers  and 
cooies  of  agreements 
with  COP  component 


"Status  cnange  forms  Showing 
date  of  referral  and  date 
of  referral  Pack  to  COP 


a  As  aoo*e 

a  Ouarterl,  OcSk  review 
of  provider  lists  and 
agreements 

a  Quarter'./  wnsite  inter- 
vwews  with  providers 


a  •tontni/,  on'. ' te  rev>e» 
of  s*^c 1-  ot  2  percent 
or  mote  at  records  0* 
participants  rtftrrtC 
dur  ft-)  uoni taring  per  od 

a  mi  'w.r  ■  "w»  which 
to  dr  m» 

a  Of .  *  <r"  Of 

■mmtn  1      ponent  status 

-epor:. 


/ 
/ 


/ 


/ 
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Manpower  Administra€iqn  (now  Employment-^wid 
Training  Administration  '[ETA]),  Program 
Assessment  QUide   (Washington/  D.C. :  U.S. 
Department  of  Labor >  April,  1974). 

Manpower  Administration  (now  Employment  and 
Training  Administration  [ETA]),  Management 
Information  Systems  Guide  (Washington,  D.C. 
%    .  U.S.  Department  5f  Labor,  April,  1974). 

Ol^pus  Research,  Self-gvaluation  o'fVgETA  Manpbwer  Programs:' 
A  Ciuide  for  Prime  Sponsors  (Springfield,  VA:  U.S.  Department 
of  Commerce,  1975)  .  -  •  n 
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Design  Management  Reporting  and  Monitoring  System/Plan 


The  purpose/of  this  step  is  to  design  your  PIC's  reporting 
systen/and  iraJnitoring  plan  so  that  it  collects  and  reports 
that  information  which  is  necessary  for  management  pur- 
poses.     This  step  includes:     defining  reporting  require- 
ments; developing  management  report  formats i  and  creating 
the  total  monitoring  plan  for  your  PIC. 


>  Reporting  Requirements  -  frequency    and  nature  of*  formal t 
reports  to  be  submitted  to  permit  effective  program 
management ^and  monitoring.  ( 

>  Management  Reporting  Display,  Formats  -  charts  for  sum- 
marizing monitoring  data  to  permit  easy  analysis  and  . 
review.  *  * 

>  Management  Assessment  Reports  -*  quarterly  narrative 
repprts  prepared  ,for  purposes  of  Council  decision-making 

^describing  PIC  program  performance  in  major  area's  of 
'  concern. 


r 


fl(  11.  K^owleoge/CaoabilUies  Rating 

Review  the  information  provided  above.  Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  this  step  on  the  scales  below. 


Knowledge 


•  Capabilities 


next  step. 


1  1 

I 

* 

J 

Excel- 
lent 

Above 
Avg. 

Aver- 
age 

Below 
Avg. 

Poor 

I 

1 

1 

1 

Excel- 
lent 

Above 
Avg. 

Aver- 
age 

Below 
Avg . 

Poor 

id-  to  read  tne  conwiu 
:ely  or  proceed  to  the 
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PMS  .PROCESS  ^OTES/ACTION  PLANNING  PAGE 


NOTES  (Points  that  are  important  which  you  want  to  remember) 


lo^jESTION^Uteins  on  which  you  are  unclear  or  want  to  get  additional 
;      information)  ^ 


4 


ACTIONS  (Specific  steps  that  you  intend  to  take  to- implement  your 
PIC's  PME  process  in  this  area) 


Steps 


Resource  s/Per sons 


Time  Frame 
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INTRODUCTION 
r  •  ^ 

As  indicated  under  the  description  of  this  step  on 
the  preceding  page,  there  are  three^tasks  involved  in 
designing  VourPIC's  management  reporting  and  monitoring 
system/plan.     These  tasks  are  to: 

•  Define  reporting  requirements 

-  •  Develop  management  reporting  and 

assessment  formats.  •  1 

;  •  Establish  the  schedule  for  your 
PIC's  monitoring  plan* 

A.     DEFINING  REPORTING  REQUIREMENTS 

The  Department  of  Labor  has  defined  certain  reporting 
requirements  which  apply  .to  your  PIC  by  stipulating  that^ 
your  Prime  Sponsor  must  submit  periodic  reports  which  include 
•  information  about  your  PIC's  operations-.     These  reports  are 
identified  and  described  below.  x 


DOL  REQUIRED  REPORTS 
INCLUDING  PIC  INFORMATION 


REPORT  TITLE 

REPORT  CONTENT/PURPOSt  L 

SUBMISSION  1-REQUENCY 

Financial  Status^ 
Report 

♦ 

V 

Collects  information  on  the  " 
amount  of  CETA  funds  that 
are  authorized,  obligated  , 
and  expended  by  the  Prime 
Sponsor.    Report  compares 
actual  with  planned  accrued 
expenditures  by  progpam 
activity. 

Quarterly 

Program  Status 
Summary  # 

Displays  cumulative  par- 
ticipation and  termina- 
tion information  and 
compares  quantitative 
program  accomplishments 
against  the  annual  plan. 

Quarterly 

■  ■ 

(continued  on  following  page) 
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REPORT  TITLE 

REPOftT  CONTENT/PURPOSE 

SUBMISSION  FREOUENCY 

Sdmmary  of  Par- 
ticipant Charac- 
teristics 

Reports  the  socioeconomic 
characteristics  of  partici- 
pants.   UUimately,  this 
.information  is  used  for 
evaluation  purposes  on  a 
local,  state  and  national 
basis. 

Ouartprl v 

i 

Program  Activity 
Summary 

Reports  th£  outcome  of 
CETA  program  terminees 
according  to  program  : 
activities  in  which  they 
participated  ancT  whether 
they*  participated  in  a 
single  or  multiple  activ- 
ities. 

Annually 

•Annual .Report  of 
Detailed  Charac- 
teristics 

Provides  cross- tabu! ated 
information  on  the  socio- 
economic characteri sties 
of  participants  served  by^ 
each  subpart  of  the^  " 
Annual  Plan.  v 

Annual  ly 

*     -  - 

Your  PIC  needs  to  ensure  that  all  the  information  necessary 
to  complete  these  reports  accurately  is  collected  regularly.  ■ 
It  is  just. as  Important  to  decide  what  additional  data  will 
be  needed  vto  allow  you  to  do  gerfonnance  monitoring  arid  how 
often  it  should  be  collected.  ~If  you  have  completed  the 
foregbing  step,  ypu've  determined  what  data  besides  that 

r 

which  is  routinely  collected  .for  DOL  reports  you  will  need 

for  monitoring  purposes.     You  now  have  to* decide  how  frequently  ' 

that  data  should  be  gathered. 

Because  the  primary  purpose  of  performance  monitoring  * 

is* to  provide  yoiyr  PIC  with  the  information  needed  to  control 
current  operations  better ,  ^tj.s  general^J^nost  useful  .to 
collect  planned  versus  actual  information  from  individual  contractors 
or  service  deliverers  on  a  monthly  basis.     This  will  enable 
you  to  identify  problems  at  that  level  early  enough  to  initiate* 
any  intervention  which  might  be  necessary.     It  is  advisable  t6 
aggregate  service  deliverer  or  project  level  data  quarterly  to  ^ 
permit  analysis  of  the  achievement  of  component  and  program    -  " 
level  performance  standards. 


ti.  ^xBEVE LOPING  MANAGEMENT  REPORTING 
*-  AND  ASSESSMENT  FORMATS  - 


.  4 


Simply  indicating  the  types^of  data  neede*  or  how  oftefi  * 
reports  should  be • submitted ,  however ,  will  not  give  your  PIC 
an  effective  monitoring  system.     To  i^e  effective,  your  moni- 
toring or' management  systdm  must:  ,  v 

mC~    present  or  display  information  in  such 

a 'banner  that  it  is  easily  usable^  "*  r 

is;  ****  > 

provide  the  basis  #f or  .preparing  reports  1  Jg 

•which  can  be  used  for  decision-making  «.  tv    ,  ,  * 

•purposes..                •  *          *  * 


permit  various  fornis-  of  data*analysis; 


Management  Reporting  Display  Formats 

In  order  to  ensure  that  you  get  management  information 
in  the  way  that  your  PIC  needs  it,  you  should  design  manage- 
ment ^reportin^  display  formats  for  recording'  information 
at  the  program,  -component,  and  service  .deliverer  or  project 
level.     In  general,  it  is  easier:  to  develop  these  formats 

by  proceeding  from  the  general  %q  specific.   'That  is,  by%     ^  , 

W  * 
deciding.  What  you  want  your  quarterly  or  summary  reports  to- 

;look  like  and  th^n  designing  the  back-up  (monthly)  reports 

needed*  to  ^compile  the   fl^p  iry  reports.  * 

Since  performance  versus  plan  is#  the  basic  technique 

for  monitoring  PIC  programmatic  sudcess,  as  a  general  approach 

to  designing  jxiaarterly  oreport.  formates,  you  should  list  ,  those 

indicators  %t6  bS  measured  employing  a    oflfia,t  similar  to  that^ 

shovm^below:    *  J 


* 


— 

Performance  Indicators 

Actual 

i  1 

Plan 

Variance 

%  t  or  "  • 

• 

.'  %  ■ '  ■  . 

* 

• 

:     (   .  • 

U^L.  ,  U.  1 

35G  - 
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Several  examples  of  forms'  for  this  type  of  performance 
•v*^ ^Versus  plan  analysis  are  provided  under  Stage  B,  "Define  PI£ 
Strategy,  Step  2.  -  Review  PIC  Program  PerformandP. An  ex- 
ample of  a  management  reporting  display  format  for  program 
level  review  of  program,  costs  i-s  furnisfied  below. 


Level':     ~  Proc^am" 


Quarter: 


^r~. — - 

PERFORMANCE  VS.  PLAN:*  COSTS  # 

INDIXATOR 

KL5ULI5 

FORMULA 

'  ACTUAL 

PLAN 

VARIANCE 

* 

.  Cost  Per  Positive 
Termination 

.  tost, Per  Entered 
Employment 

Tot.  Accrued  Exp. 
Tot.  Pos.  Terms. 

Tot.  Accrued  Exp.  . 
Tot.  Entered  Emp. 

• 

.  Cost  Per  Indirect 
Placement 

Tot.  Accrued  Exp. 
Tot.  Indir.  Place. 

An  example  of  a  format  for  EGS  projects  follows. 


EGS  Projects 


-Quarter: 


ERIC 


PROJECT 

ACTIVITY 

INDICATOR 

Plan 

ACTUAL 

VAR. 

1 

Labor 
Market 
^Analysis 

Completed 

employer 

survey 

Usable 
responses 

100  by  end 
of  February 

* 

* 

s 

Linkages  - 

Relationship 
establ ished 
wHti  economic  t 
development 
agency 

Parti  ci  pa-, 
tion  in 
*  planning 
process 

Joint  planning 
in  February 

:  j 

t 

i 

• 

\ 

Creation  of  ' 
training  pro- 
gcam  for  new 
jobs  created . 
thVough  eco- 
nomic develop- 
me^^effort 

.Completion 

date  s 
.Acceptabil ity 

to  employers 
.Adherence  to 

state  guide- 
•  lines 

/ 

m 

..Developed  by 
end  of  March 
i  .AH  employers 
*will  endorse 
.S^ate  Board 

will  review 
^  and  approve 
™  asrjmeeting 
State  guide-y 

9  , 
• 

m 

i 

* 

V 
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Data  Analysis  •  > 

You  should  not  restrict  your  data  analysis  to  one  type, 
In  designing  your  management  reporting  formats  you  should 
develop  forms  which  Sllpw  you  do: 

•  Trend  Analysis;     study  of  performance  over 
time  to  see  if  it  is  increasing,  declining, 
or  stay ing  the  same .  , 

m  •  Comparative -Analysis;     study  of  th#  relative  % 

performancte  of  two  or  more  components,  or 
^  service  deliverers. 

Examples  of  forms  which  cfoul^  b£  used  for  trend  and 
^^comparative  analysis 'are  presented  on  tfte  next  tkree  "pages. 
Analyzing  data  using  these  techniques  will  enable  your  PtC 
to  secure  a  mgpre  total  picture, -to  draw  more  valid  and 
.   reliatjj^  conclusions  regarding  yo.ur  prograrf^ s'  performance, 
and  to  prapoint  more  exactly  those  areas  in  .whfi^h  corrective 
acti^ui/or^^ogram  improvement  -^s  necessary. 


\ 


r 


3?  8 


ERIC 


111^247 


t 


^tep  3 


TITLE  VII 

PERFORMANCE  INDICATOR  TRENDS:     SIGNIFICANT  SEGMENTS  PERCENTAGE  OF  PROGRAM  ENROLLMENTS 


SIGNIFICANT  SEGMENT 
ENROLLMENT  CATEGORY 

 fc  !  2  

1st  Quarter 

2nd- Quarter 

3rd  Quarter       j;       4  th  Quarter 
!|             FY  80  * 

* 

Pl.% 

**]%-of 
Act.%  1  Dev. 

* 

pi.% 

Of 

Act .  %  -i  Dev. 

» 

PI . % 

•  *. 

Act .  % 

%  of 
Dev. 

1  pi.% 

Act.%  1  Dev. * 

Mal4*  • 

'! 
I 

I  t 

1 

i 

j  1 

1 

i 

 :  

Female                                     \  . 

f\ 

-■  I 

• 

i 
1 
i 

19  &  Under  f 

/ 

1      ■  i 

• 

r  

20-21 

"  1 



i 
i 

• 

22-44 

!  1 
i  i 

• 

45-54-   /\ 

4  1 

\  '  ' 

i  i 

!  4* 

:  1 

55  &  Over  . 



[  ^ 

1  

1  i 

i  ' 

* 

i 

1 

W.   -   (N.H.)  V 

 i 

1  . 

1  - 

  %  

'    B.    -  (N.H.) 

Hispanic    •  ***  j 

1— ;  

\ 

i 

i 

I— 

1  :  

1  :  ;  ; 

A.I.   6  AN.  ! 

_  :  , — :  :  1 

> 

V 

 1— — 

a.  ^p.i. 

1 

•t 

1 

I 

00 
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♦Plan  percentage:  Use  demographic  tables'ih  Annual  Plan  or  compute  from  PPS  (Part  ^/A-L) 7 (pPrt  *A) ; 
**Actual  percentage:     Compute  from  QSPC   (sig.  $ecj. ):» (Total  partic.  ei^olled)  ■  *  * 


* 


f 


COMPONENT: 


M Classroom  Training 


Quarter 


SERVICES  DELIVERERS 

'enrolled 

TRNG.  EXPENDITURES  , 

POS..  TERMINATIONS 

ENTERED  EMP. 

• 

Plan 

Act. 

Var. 

Plan 

Act. 

Var. 

Plan 

Act. 

Var. 

Plan 

Act. 

Var. 

Project  A 

Project  B 

Project  C. 

< 

-  deliverer  1 

-  Deliverer  2 

* 

• 

»  r 

 % — 

I 

rO 


Another  indicator  frequently  used  in  this  area  is'pre-CETA  earnings  vs.  post-CETA  earnings, 
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Step  5 


I 
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TITLE  VII 

PERFORMANCE   INDICATOR  TRENDS*:  EXPENDITURES* 


4th 


1st 


2nd 


3rd 


4th  ! 


PROGRAM  COMPONENT  * 


Quarter    !    Quarter        Quarter    j    Quarter    j  Quarter 


*  Enter  percent  of  program  funds  expended,  compared  to  planned  budget. 
Ref:     Financial  Status  Report. 


i 

CLASSROOM  TRAINING 

i 

!  • 

% 

% 

% 

• 

% 

—4  

OJT 

* 

|— 7  

 .  

SERVICES  TO  PARTICIPANTS 

a 

EMPLOYMENT  GEN.    SERVICES  / 

—    .  7 

■ 

  1 

#  • 

OTHER 

1 

. 

f  ~  ;  '  

OTHER 

•S  ! 

TOTAL  EXPENDITURES  * 

t 

■  * 

i             s  * 

Upgrading 

i 

!  i 

Retraining 

i 

i    ■  • 

i     i:    1  • 

n 
o 


H 
03 


z 

D 


ERLC 


363 


364 


\ 

Management  Assessment  Reports 

The  data  produced  by  your  PIC's  reporting  system  and 
summarized  on  your  display  charts  is  raw  information  for 
the  management  asses^Jent  of  your  ^C.     In  order  to  be 
most  njafeil ,    that  information  should  be  organized  into 
formal/assessment  reports  which  can  be  u^ed  as  a  primary 
sourceVor  making  programmatic  and  management  decisions. 

general,  your  PIC's  management  assessment  reports 


X^st^uld  provide  the  following  types  of  information: 
4 

•  A  description' of  the  current  status 
of  the  component/area  being  reported 


•  Identification  of  'performance  that  has 
deviated  from  plan 

•  Analysis  of  the  current  situation  and 
analysis  of  the  cause  of  deviations  * 

•  Identification  of  those  areas  that 
require  corrective  action 

*      •  Recommended  actions. 

Usually,  management  assessment  reports  should  be  prepared 
quarterly  to  coincide  with  DOL  required  dat^  preparation 
and  yoar  PIC's  internal  managj^ment  performance  charting. 
However,  special  circumstances  of  your  PIC  may  indicate  the 
ne#£L  for  more  or  less  frequent  reports.     Some  possible 
^parterly*  reports  which  might  be  prepared  by  your  PIC  'are 
listed  below. 

Possible* Quarterly  Reports t     The ^ following  quarterly 
management  'assessment . reports  would  provide  your  PIC,  with 
a  significant  amount  of  the  dat^a  needed  to  monitor  and 
manage  performance:  ' 

•  Descriptive  Analysis  of  Overall  Program 
^Performance^ to  Date  -  describing  the 

current  status  of  the  program  with  em- 
phasis on^fiferf ormanc'e  in  relation  to 
planned  objectives. 


3S5 
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•  Component/Project  fceVel  Analysis  of per- 
formance to  Date  -  providing  data  on  per- 
formance of  specific 'components  and  projects 
highlighting  areas  required  by  formal  correc- 
tive action  plans. 

t  Descriptive  Analysis  of  Participant  Charac-  .  % 
teristics,  -  describing  program  performance 
^    ""in  recruiting,  enrolling         serving  sig- 
rjificant  segments. 

)     '  f 

•  Assessment  of  Program  Responsiveness  to 

.    Identified  Needs  -  providing  an  analysis  of 
the  PIC's  program  design  in  meeting  the 
needs  of  businesses  and  target  groups 
,  ,  within  the  local  community. 

As  in'  the  other  areas  of  the  PME 'process,  however,  you 
should  make  .the  determination  of,, what  your  particular  local 
needs  are  and  then  design  your  management  assessment  reports 
to  meet,  those  needs. 


C.     DEVELOPING  MONITOfclNGl  PLAN  AND  SCHEDULE 


Having  es.tatt-ished  your  PIC's  reporting  requirements  'and 
developed  your  management  reporting  and  assessment 1  formats  ,# 
your  last  task  is  to  develop  a  time-phased,  monitoring  plan 
for  the  year.     That  plan  should  list: 

« 

#%  .    all  tasks  and  activities  to  be  completed 
»       the  periods  of  time  in  which  they  are  to 
be  completed 
due  dates  for  when  specific  reports  or  * 
products  are  to  be  delivered.    > * 

£ome  /of  the  tasks  and  items  ^hich  should  be  oif  your  time-phased 
calendar  include:  * 

Planning 

Defining  operational  control  process  (monitoring 
method (s)) 

^Assigning  management  review  responsibilities  * 
Defining  management  information  needs  * 
Identifying  objectives  and  indicators 
Defining  reporting  requirements 
Developing  data  display  formats 
Developing  data  analysis  procedures 
Developing  management  assessment  report  formats; 
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Operating 

•  "  Designing  specific  monthly  reports  >- 

•  Collecting  data 

•  Generating  monthly  reports  (program  operators)  * 

•  Summarizing  data  monthly  on  display  charts 
*•        Analyzing  data 

•  Aggregating  data  quarterly  -  program,  component, 
all  service  deliverers,  levels  , 

•  Formulating  findings,  conclusions,  and  recommen- 
dations 

•  Preparing  and  issuing  quarterly  management  assessment 
reports. 

As  noted  above,  other  items  which  should  go  on  your 
time-phased  monitoring  plan  are  specifier  due  dates  for 
products  or  services  to  be  delivered  by  contractors/.  Noting 

r 

these  dates  and  interim  targets  is  especially  important  for 
the  EGS  projects  to  be  run  through  your  PICr,  ' 

One  final  task  which  is  an  important  part  of /your  moni- 
toring plan,  btft  which  can't  be  given  definite  tonnes  on  your 
time-phased  calendar,  is  "take  corrective  action/"    This  task 

v  / 

should  be  scheduled  as  necessary  based  upon  youi?  monthly 
ahd  quarterly  analysis  of  data.  / 

In  conclusion,  your  PIC's  monitoring  plan/provides  the 
time-phased  work  schedule  to  evaluate  program ^progress  against 
projected  c'ompletioji  dates.     In  Addition,   it  is  the  necessary^ 
tool  for  determining  the  type  and  level  of  re/sources  necessary 
to  implement  your  PIC's  monitoring  program.  / 

'  \ 


ID  1 
I  REFER! 

NCCS 

Manpower  Administration   (now  Employment 
and  Training  Administration   [ETA]),  Program 
Assessment  Guide  (Washington,  D.C.:  U.S. 
Department  of  Labor,  April,  1974) . 

Olympus  Research,  Self-Evaluation  of  CETA  v 
Manpower  Programs;     A  Guide  for  Prime 
Sponsors   (Springfield ,  VA:  U. S .  Departme n t 
of  Commerce,  1975) . 
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DEVELOP  EVALUATION  PLAN 


Purpose;       To  establish  a  detailed  plan  and  procedures 
'for  evaluating  your  PIC's  pcpVram,  t 


1.  Establish  Scope  and  Nature  of- 
Evaluation  4  , 

0 

2.  Assign  Evaluation  Responsibilities 
and  Establish  Procedures 

3-     Construct  Evaluation  Research 
Design 

4.     Create  Research  and  Data  Gathering 
Instruments 
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^tCp  ^       %  Establish  Scope  and  Nature  of  Evaluation 

The  purpose  of  this  step  is  to  review  your  PIC's  program 
operations  plan  to  determine  the  exact  scope  and  nature  of 
your  PIC's  evaluation  effort.     There  are  three  fundamental 
types  of  evaluation  which  your  PIC  can  decide  to  undertake 
based  upon  this  analysis:    outcome  or  results  evaluation; 
relative  effectiveness  evaluation;  and  impact  evaluation. 

f 


Q  Key  Conceots 

Outcome  Evaluation;     assessment  of  actual  performance  v. 
plan  for  fiscal  year . '  * 
Relative  Effectiveness  Evaluation:     comparative  analysis 
of  projects,  services  delivered,  or  program  components  ^ 
undertaken  to  ascertain  the  relative  meri-ts  of  ^each. 

Impact  Evaluation:    measurements  to  determine  the  extent 
to  which  the  program  has  brought  about  participant  change, 
institutional  Aiange,  or  otherwise  had  an  effect  upon  the 
community  in  which  your  PIC  operates. 


^a^KmwIedge/CapabnUies  Rating         CZ>~  ,t 

*  Review  the  information  provided  above.     Rate  your  PIC's  knowledge  "and 
capabilities  with  reference  to  this  step  on  the  scales  below. 


•  Knowledge 

•  Capabilities 


1  

I 

1 

1  ' 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

I 

i 

 1  

.  1 

Excel- 

Above 

Aver- 

Below 

.  Poor  p 

-  lent 

Avg . 

age 

Avg. 

<  Ba^ed  upon  your  ratings,  decide^Whether  ydu  need  to  read  the  content 
'    ^information  on  this  step,  which  follows  immediately  or  proceed  to  the 


next  step.. 
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.  f ME  PBOCESS  NOTES/ACTION  PLANNING  PAGE 


QjEsriGlss'Trtem^OT  want^  to  get^ additional 

'  information).     •  f  '  t      •  '>  • 


5crToN^Spec7?Tc^t5pT"tHat  you  intend  to'^ke  to  implement  your 


'  PIC's  PME  process  in  this  area) 


9 


StepI  ' 


Resources/Persons 


Time*  Frame 


\ 


/ 


/  ■ 
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INTRODUCTION 


1  rr  1 

CD 
t3 


*    *   /£n  evaluation  provides  the  means,  *t  ttm  conclug4o^o£~,  1 — 
an  ^entire  performancre  period,  to  asseq^  the  ef f ect  of  fyour 
P>JC's,  programs  and  activities.'    Your  PIC's  evaluation  can 


serve  a.  number  of  purposes,  .including:"       »  • 

•       to^  identify  problems  and  deficiencies  . 
in  current  operations  • 

to  estimate' the  effectivdhess  of  your  ^J*/* 
prpgram  041  achieving  specified  results0^ 


* 


conate  alternative  sti^t^fries  arid  " 


to 


.  tc^refine  planning' assumptions. 

•     ;  to  measure  gaps  between  planned  out- 
comes and  results.  • 

I  ,  %  V  •  -  * 

H    Most  importantly,  evaluation  data  can^ptovide  feedback 

►upon  your  PTC's  program  planning,  operations,  and  maaagement. 
This  C^JF  used  to  improve*  f irfure 

•       programmatic  performance  ^ 
r>     •       organization  and  operations  *  ^ 

p     9       planning  and  management. 


V. 


frypes  of  Evaluation 

"  ^.^c^make  your  evaluation  most  useful-,  it  should  be  planned  * 
at  the  same  time  that  your  pic's  programs  are  being  placed. 
In  fact,  your  PIC's  program  op^rat^bns  plan,  ofcpectiv6s,*;  and  , 
performance*  standards  repr^seni;  the  primary  sources  for  deter- , 
mining  precisely 'what  "to  evaluate.    The  purpose  of  this#step  is 
to  ireview  that  plan  to  identify  the  scope* anfl" nature  of  your 
evaluation.  ' 

•  There*,  are  three  primary  types  of 'evaluation  whiph  your 

PIC  can  decide  to  undertake:.  .'.  ^..J,  '    » *m 

\  *'  '  *  •  •  * 

q#tcome:    assessment  of.  performance  against',       ■>  ^ 

standards  set  at  the  beginning  of\£g|H;armance,  * 

of 


period  fo  determine  the  degree  of -lachiovemon^t 


ERIC 


•1  A 
4 
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^Relative  Effectiveness;    comparative  analy- 
sis  of  projects,  .tactics,  service  delivered, 
£n<Vor  program  components  to  ascertain  / 
differential  benefits  and  merits  of  each.  ✓ 

' Impact ;    measurements  to  determine  the  extent 
to  which_the  PIC 1  g^locjratt  has  brought  about 
^     participant  change ;~3mstitutional  change, 
"   and/or  .accomplished  As  program  goals. 


The  Evaluation  Decision 

^  Your  PIC  should  decide  what  formCs)  of  evaluation  to 
implement  based  upon  its  decision-toaking  needs  and  Uie  resources 
you  have  available.  'Yqu  should -at  least  do  outcome  evaluation 
in  those  areas .which  you  consider  to  be  of  high  priority  for 
program .  success  arid  \fy  which  you  need  information  forj^- 
planijlncj  purposes.    You  should  supplement  this  basic  ^WRua- 
vtion  approach  with  -relative  ef fectiveness~and/or  impact 
evaluation  measures  as  you  deem  appropriate  arid  necessary. 
The  pages  which  follow  present  information  and  describe  measures 
your  PIC  might  Jemploy  in  implementing  each  type  of  'evaluation. 

•    •  ;   L  '  . 


■Si 


\^y*    JX3ING  OUTCOME  EVALUATION 


\t  a  minimum,  your  PIC  wil^L  want  to  evaluate  outcomes  Or 
results^gainst  those  performance  standard^  which  you  set  in 
•  developing  your  program  operations  plan.     This  evaluatiQn  can 
be  dorfe~ very  easily  by  directly  contrasting  actual  versus 
planned  performance  * for  each  area  in  which  you  set  performance 
%  standard?.^.     .  #,  , 

As  noted  earlier,  on,  the  participant  side  of.  the  ledger-, 
much  of  the  quantitative  data  you  need  ta  permit  this  type 
of  ^sserfstoent  can  be  retrieved  from  tHree* reports  -r  Program 
Status  Summary,  Summary  of  Participant  .Characteristics,  ^ndJ 
Financial  Status  Summary  —  prepared  quc^erl^r  and'afrhually 
j£p£  the  %epartitent  of  Lai>or.     ItM*i],l  t>e\sbij^wha,t  mbre  diffi- 
cult to;  attain  djfct^i  in- tti^  EGS  <are$T    Howev^b,  *if  you  tfiave  s£t 


* 


performance  standards  for  <each  EGS  activity >  then  you  can 

*  „  V 

depign  a  reporting  system  t?o  collect  the  necessary •  data  to 
tell  you  whether  the  standards  were  met. 

An  important  point  to  remember  is  that  not  all  of  your- 
^ PIC's  projects  or  ^services  can  or  should  be  evaluated.  -  Given 
limited  financial  and ,labor  resources,  you  should  restrict  your 
evaluation  to  those  projects  and  program  components  which,  in 
your  opinion,  are  most  demanding  of  assessment.-  In' general, 
a  project  should  be  evaluated  if  it  "has  any  of  the  following 
characteristics:  ;  " 

•  *  it  is  new  "  *  V  ^ 

m'       it  is'ongoing  and  has  never  been  'assessed 

•  .  .  it  is  in  £m  area  in  whiclj.  a  latge  invest- 
•   „  *       ment  of  resources  and/or  manpower  has  been 

made  4 

•  it  "is  experimental-  in  intent  and  design. 


•  Ba'sic  techniques  arid  fo/mats  for  reviewing  plan  vs.  per- 
formance are  described  in  detail' under  "Stage*  B.     Define  PIC 
Strategy.:  -Step  1.  ,  Assess  PIC  ^Program  Performance,  "  pp.  111-65-96. 
,As  noted,  an  Outcome  evaluation  will  probably  satisfy  most  of 
your  PIC fundamental  evaluation  needs.     Thev  other  forms  of 
evaluation  can  be  employed  to.  enhance,  your  PIC's  decision- 
making capabilities. 

'i     B.     DOING  RELATIVE  EFFECTIVENESS  EVALUATION 

'    Asse^|^ig,  the  relative  effectiveness  of  the  components^, 
^nd'effectrs  of  your  PIC-'V  program  is  especially  useful  for 
replannig  purposes.     Data  on 'relative  effectiveness  can- help 
your  PI(>  dgc^de* what  to  continue,  "fetop,  increase,  or  decrease. 
This  type  of  evaluation- enables  you  to  assess  the  relative 
rilerits,  deficiencies,  arjd  coats  of  dif f erent^programs  or 
comp^nent^/ regardless  o/,tJfeir  apparent,  c^i^Simi^arity.^ 

K 


< 


^  comp^nentp^  re 
R  it 


On  th^  participant  services  side  of  your  program,* if  your 
PIC  has  incijrporated  indicators  for  terminations  and  cost^  • 
such  as:  *  * 


Terminations 

Positive  Termination  Rate 
'   Entered  Employment  Rate 
*  Indirect  Placement  Rate 

Private  Sector  Placement 


Costs 


Cost  Per  Positive  TermihatTom  r 
Cost  Per  .Entered  Employitentj  ^ 
Cost  Per  Indirect  Placement 
.  C<fst  Per  Privat^  Sector  placement 

then  yo\i  already  -have  data  reported  in^such  a  fashion  that  it 
can  be  used  in  a  simplified  comparison  of  relative  effectiveness 
*  of  components  and  projects*   'You  can  deveiop  forms ^patterned 
after, those  on  pp.  m-79-82  to  permit  that  type,  of  analysis.- 

^Relative  effectiveness  comparison  on  the,^S  side  of  your 
PIC's  program  will  be.  somewhat  more  difficult  ™ven  the  great  Js^ 
variety  of  projects  which  can  be  iijrtia^ed  in  this  area.     How—  A 
•ever ,  ' possible  bottom  line  indicators  for  Avany  of  these  projects', 
regardless  of  their  primary  method  (e.g.,  linkage*  economic 
deye lop^ient )  ,  might  be:*  *  >  * 

•  jobs  created  or  vacancies  developed; 

•  jobs  filled  by  CETA-eligible  par- 
ticipants; 

j         •        average  wages  of  jobs  secured. 

If  these  indicators  can*be  associated  with  your  PIC's  EGS 

.projects,  them  yop,  could  Vp£ly  a  relative  effectiveness  per- 
spective by  us%njf:th%se  ihdyAtors  alone  or  in  conjunction  with 
the  costs  associated  with  tha  parjaealar  projects  to  permit 

*  effectiveness  comparison  &etwegtf  and  among  projects. 

For  example,  if  yqrfr  PIC  initiated  a  phone  solicitation 
of  lpca*-r-enipl9yers  to  create  Job  vacancies,  and  also  initiated 

*  an  -economic  development  project  "With  those  local  construction' 
companies  to  creWe  jab  vacancies,  it^would  be  possible  to 

'  compare  the  two  iprojects  by  using  simple  measures  such  a$: 


Costs 


*  Jobs  created 
Jobs  fi] 


» Cost*  per  jol?  created 
Co£t  per  job  filled 
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However,  doing  relative  effectiveness  analysis,  relying 
solely  on  a  -  straightforward  comparison  of  results  on  the  singl 
quantitative  indicators  cited  above,  could  be,  very  mi^Leading. 
To  make  relative  comparisons  and  analysis  more  meaningful,- 
yoii  need  to  try  to  make  the  concept,  of  effectiveness  ta  be 
Applied* in  comparing  projects  and  components  as  appropriate, 
comprehensive,  precise,  and  rigorous  as  possible.     Some  * 
possible  methods. which  you  might  employ »to  achieve  these  ends, 
relying  primarily  on  data  whifch  can  be  retrieved  from  your  ,< 
PlC's  'reporting  system*, ,  inclucfc: 


0  m.       Effectiveness  Indexes 

^  •*       Cost  Effectiveness  Indexes 

•  •      Composite  Effectiveness  Indexes. 

-  /~ 

Effectiveness  Index  * 


-r- 


'  An.  effectiveness  index 'may  be  loosely  defined  as  a 
.performance  measure  which  combines  two  or  more  measures  of 

effectiveness  to  derive-a  result  (i-.e.,  effectiveness  index) 

r  \  • 

stated  in  numerical  terms.     One  of  the  most  common  effec- 
tiveness indexes  is  the  Placement'/Wage  Index. 


1.     P3racement/Wage  Index 

The  Placement /Wage  Index  is  defined  as-: 

/  V 

(Placement  Rate)  x  TAver^ge  Hourly  Starting  Jflage) 

For  example',  if  the  placement  r*ate  of  a  Wgldirdp  Program  was 
60%  an^  the  average  starting  wage  was  $3.63,  'the  index  score- 
wquld  be : 

*  *  * 

,    .60-  'x     3.63-     2.18  •  ^ 

On  ' the  other  hanfi,  if  the  placement  rate  of  a  Hospital  Attefl- 
dant  Program  was  93%  and  the  average  starting  Wage  was  $2.25/ 
the  'index  scor^  would  be:  ^     v}  * 

1    .     w     6        :         .93    $    2.25-'  =     2.09  * 

*The  higher  th^  scope 4  the  better  the  performance  according  tqs 
the  index .  \       '  1  <  •  .  *  * 


r 


. -The  Elacemenf^Wage  Index  caji  be  used  to  rank  a  number  of 
-  projects. according  to  their  performance  on  the  index  in  ordrf 
-to  facilitate  their  comparison.     However,  that  ranking  in 
; itself  Abes  not  enable  your^  SIC -to-nake  a -decision.  Rather, 


,*  .    >   ^  nterely  cjives  your  PIC.  a  usefuj.  perspective  for  comparative 
.analysis  aiggng  $il(j^er  projects  x?r  components. 


/ 


\      .         •  v' 

'   k       f;  /  V -        2.  .  uj&ng  tiie  Placement/Wage  index 
"  *  '  *  . ,         '      *    .  /    m  Decision-flaking 

\     ;  •      -     ;  >  •*% 

•  *  .fhe  Placeirfent/jyage  Indeuf  allows your  PlC  to  look  somewhat 

-ill),  /itore^ob^tet^eiy-  an^  thoroughl^at  program  performarfee  and  to 
*deqid^.what#  J^de-of  f-s  *yotf  wai^Pto  make.     The  index- enables 
yotr  tj6^no£  'dijccwar^t  *aot omaticarlly  those  projects  which  have  ,J 
.•*•;■  relatively  'J.ow'"  placement  rates  but  .at  tiigher  salaries,  or 
ovtfrestibate  th'4  poteiifiai  or  ^ntftribution  of  those  projects 

but  in  relatively  *  low-paying 


ment  rates  1 


7whioh  result -iij. 
*?bs>      *?/  - 

,vt>  -     The  decision  wfciclT  /yopr  PIC  has  to  reach  iS  whether  a  * 

**-  •         /'".'*.*  -      *     v        -  I  V* 

project  -with^an  Average  on  below  average  plpc^ment  rate  leadrflg 

*  •  -  «*  " 
to  steady.*jobs  wi-fetiJrigh  wages  is  preferable  tQ  one  with  good 

ng  t*/  jobs  wibh  low  vageLS."'  you  would 
undoubtedly' make  this  decisior^T>ased  upon  1:he  current  socio- 
economic needs  in  the  cfetomunity.,. 

-       The  Placement/Wage,  Index  is  just  one  ^o'f  mahy -.measures 
that  can  be  used  to  lodk*'<t  yi^ur^ PIC's  per f prmancf* pn  th? 
participant  side.     Although  "this  index. and  others  ld.ke  it 
observe  .some  data  *nd  ^re  rtofc.  completely^fall-safe  or  totally 
r^liafile  means,  for  evaluating*  effectiveness, '  they  do  aftlow 


ranci$e  ^nd  simple- 


present.  dat£  in  a, 
nner 

;  -       .  ,    '  '      "7        '    '  '        -  /  *• 
^sta^xhh  iti  aft  exilic  it  'manner  -those 
mp^t  imj^ftant  factors  ori  criteria 
^hich  yaw  JPIC  Vants  - to:  ^pply  in'eval- 
iating  ^prenects  and  0O54^ne^ts.  \ 


,  -  J- 

Cost  Effectiveness  Indexes 

One  of vthe  factors  which^ur  PIC  will  probably  consider 
worth  knowing  from  an  evaluation  standpoint  is  the  relationship 
of  the  cost  of  comppnei>ts  or  projects  to  their  effectiveness 
in  achieving  planned  results.     This  can  be  determined  by  > 
developing  cost  effectiveness  indexes  which  combine  costs 
with  effectiveness  measures.     There  are  two  principal  methods 
for  determining  post  ef fectiver>ess:#    the  simple  division 
method,  ind  the  proportionate  mfethod.  ^ 

1.     Simple  Division  Method 

The  simple  division  method  of  determining  cost^-ef fiective- 
ness  is  to  divide  the  average  unit  cost,,  such  as  average  cost 
per  earollee,   into  a"mea~sure  of  results*or  ef f ectiveness7~  " 
such  as  % the  Placement/Wadfe  Index,  and  then  multiply  the 
quotient  by  a  Number  (N) ,  such  as  1000,  to  move  the  deci^ 
back  and  add  presentation.     This  method  is  summarized  ii 
the  formula  below: 

Placement/Wage  Index  {Effectiveness  Measure)  ,  x  1000  = 
'         Average*  Cost/EntolXee  (Cost  Measure)        5  ^ 

•Cost  Effectiveness  Measure  ■  %  '  ^ 

For  example,  if  Program  A  had  an  average  cost  per  enrollee 
of  $1500  and- a 'Placement/Wage  index  of  193;  and  Program  B  , 
had  an  average-  cost  p^E  eprollee  of  $1200  and  a  placement/Wage 
index  of  1(83 ,  their  cost-effectiveness  indexes  would  be 
calculated  as  showj^below: 

'll  x  ■    1291       -     iMo   x   1000   '  153 

As  the  figures  show,  utilizing  this  cost'-eff ectaveness^ 
measfciretaent  t  approach,  Program  ^  would  be  seqn'  as  morec^t' 
.effective  than  ^Program.  A.     This  is 'to  be  expected  in  thiSN 
approach  where  unit  cost  serves  as  the  denominator ,in  J 
cost-ef fectiveness,  ecjuation,  and  where  the  impact  of  tl 
unit  costs  upon  effectiveness  will  vary  considerably  according 
to  the  size  of  that  unit  cost/"  • 


■i 
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2.     Proportionate  Method 

The  second^ method  qf  determining  cost-effectiveness  is 
designed  t£>  produce  an  idpact  upon  the  effectiveness  treasure 
which  is.  proportional  based  upon  the  size  of  the  unit  cost 
in  relation  to  another  number  rather  than  absolute*  _The  way 

this  isv  done  is  by  eistablishing  a  cost  factor  in  which 

averag£  unit  costs  become  the  numerator  and  an  appropriately 

chosen  number  -  a  dollar  figure  normally  double  the  unit 

cost  of  the  most  expensive ^project  -  becomes /the  denominator 

as  shown  in  the  formula  below:  " 

-    ^  „  \  ,       C  (average  unit  cost) 

Cost  Factor  m  1  -  -n — 7  — ■ — r  c — -t- 

1L. (appropriate  number)       -  , 

r  * 

This  factor  can  be  used  to  reduee vthe  effectiveness  measure 


by"  a  constant  proportion  of  its  value  for  every  $1000  in 

un^t  cost.     This  results  in  equal  change  factors  for  equal 

increments  of'cosf. 

Let'*s  apply  thisytechnique  to  look  at  Programs  A  and  B 

which  we  discussed  earlier.     For-  the  sake*of  an  ,aftalysis ,  let 

say-  that  the  N  -  appropriate  number  -  or  denominator  in  the 

equation  ^should  be  .$20,  000 .  00 : 

*  Cost     *>    '    „  „  „      '  Cost: 

Program  -  ,  Eff.lnd.  \      *r°9*™  •  Eff.Ind. 

1  "  207000  X  193     =     179  '      1  "  20~000     X     i83     "  172 

*****      -   '  «  '  - 

As  the  results  sjiow,  using  this  formula,  Program^A  has  • 

a  more  favorable  cost^ef fectivepeSs  ratio  than  Program  B. 

ThiB  is  the  opposite  of  the  results  obtained  using  tfie 


simple  division  'itiethQd.  *  , 

Which  measure  .is  correct?    Is  Program  A  or  B  preferable? 
The  answer  is  it  all ^depends Z  It"  depends  on  what  your  Pl6^ 
deems  to        the  best  way  to'  assess  cost-effectiveness,  the 
results  you  obtain  using  that  approach,  the  importance  you 
cftoose  to  assign  td  the  index  rating,  and"  the  interpretations 
yoU  give  to  the  data  y.6u  obtain. 
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^  j    '      •  *•        *  *  - 

As 'noted  ear lifer  Regarding  the  Placement/Wage  Index,  no 

m  v 

effectiveness  index  gives  youiSnswers.     Such  indexes  are  useful 
fbr  ranking  and  comparison,  bUt^they  do  not  furn^h  judgment's. 
It  is  your  responsibility  to  consider  alternative  indexes  and 
^hen^ecide  whichpwill  4ye  most  consistent  the  the  perspective 
of  yo%:  PIC,  and  will  provide  you  with  that  information  which 
will  be  most  useful  for  decision-making  purposes.  % 


Composite  Effectiveness  Indexes*  .  * 

A^asic  problem  of  both' the  simple  combination  effec- 
tiveness measures  and  the  two  cost-af  f ectiveness.  measures 
discussed  to  this  point  is  that  they  are  relatively  restrictive 
in  their -scope  and  do ^not  provide  a  tbtal  picture  of  the 
projects  or  Components  being  compared.     One  option, that  your 
PIC  might  want  fco  consider  inforder  to^et  a  broader  per* 
spective  on  projects  or  components  is  composite  effective-, 
ness  index  composed of  several  factors  weighted  according  to 
their  importance-  , 


*  1.'  The  Performance  Rating  System 

✓  One  very  -inclusive,  composite  index  is  that  developed  by 
Don  Menzi  of  the  New  York  Planning  Council.     This  index  is. 
called  the  Performance  Rating  System.'    Although  it  was  developed 

*'for  systematic  evaluation  and  comparison  of  programs  funded 
under  the  categorical  manpower  system  existing  before  CETA,  it 
is  still  .applicable  to  many  of  th£  paf ticipant-serving 
projects/iomponenjzs  of  your  PIC.     It  is -\particularly  appropriate 
for  those  projects  where  a  contractor  or  operator  has  unshared 
responsibility  for  participants  "f rom  entry  into  the^  progr'am '  to 
exit.  4  -  • 

The  Performancd^jjating  System  uses  a  cost  factor  x  multi- 

-  factor  effectiveness  measure  to^etermine  effectiveness.  %The. 
basiT:  formula  for  thfc  system  is: 

Cost  Faqtor^-rt    Multi-factor  Effectiveness  Measure 
=    Effectiveness  Index 
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The  cost  factor  in  the  Menzi  system  is: 


1  -  Average* Cost  per  Placement' 
^Jp5o  *  


The  multi-factor  effectiveness  measure  is  a  Weighted  average 


based  on  six  factors: 


V 


s 


i. 

2. 
3. 
4. 
5, 
6. 


Employment  Rate  (weighting'*  40%) 

Follow-up  Completion  Rate  !( weighting  5%) 

Retention  of  Job' (weighting  =  5%)  • 

Wage  Index   (weighting     dX)^)  , 

Target  Group  ihd^x  (weighting  £20%) 

Actual  vs.  Planned  Placemen^.  Objectives 
(weighting  =  20%)   '  ■ 


The  Target  Group  Index  is  developed  Rasing  the  estab— ■ 


) 

lished  priority,  target. groups  of  the  planning  council. 

J  v 

Wage  Index  measures  tt\e  perpentage  by  wRich  t^he  post-, 
training  wage  exceteds  the  accepted  Poverty  Ir^dex  for  a 
family  of  four.   ,  ' 


The 


2  .     Another  Composite  ^£^ex 

An  . e^TOiale  ofi^another' composite  index- developed 

►laying  the  Menzi  Model 


by  a  CETJI  Prime  Sponsor, 
,  to  measure  the  relative  effectiveness  Df  components  and 
contractors,  is  described  briefly  below.  # 

This  rating  inderx  consisted  of  five  factors  Veighted 
as  indicated: 


*  • 


Weighting 


Factor 


.  Planned  versus  actual  performance 

.  Extent  of  service  to  those  "most 
,   in  need" 

m 

.  Extent  to  which  participants 
complete  component  activities 

.  Extent  to  which  participants 
achieve  primary  outcdme  specified 
for  component       /  < 

I  m 

.  Cost-effectivene^  (ccrst  per 
specified  outcome) 

4 

Total  Score  - 


Component 
10% 


Contractbr/Service 
Provider 


30% 


35% 


5% 


100% 


15% 
10% 

35% 

30% 


ido% 
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Each  factof  was  assigned  possible  points  which  cou^d  * 
be  scored  on  it^agcording  to  its  percentage  weighting  as 
shown  in  the  chart  below: 


FACTOR 
1 

COMPONENT 
POSSIBLE  SCORE 

•  CONTRACTOR 

 MAXIMUM,  ' 

POSSIBLE  SCORE 

Planned  vdL 
Actual  Index 

1.5 

* 

Service  to  Those  A 
in  Need  Pndex  - 

1.0  ' 

1.0 

Completion 
Index 

3:0 

3.5 

Outcomes  1 
Index 

3.0 

Cost  Effectiveness 
Index       .    4  .  / 

1.0 

All  / 
Factors  J 

lo.o 

/      1 0 '  °, 

Then,  rating  scales  and  criteria  for  scoring  points  on  each 
factor  were  developed.     For  example,  the  criteria  and 
scoring  procedure  for  the  Performance  versus  Plan  factor  was: 


Criteria 

Met  100%  of  Plan 
Variance  of.  0  to  +  10% 
variance  of  ipore  than  +  10% 


Scoring 
Component.    -Contractor  • 


2,0 
1.5 
-0- 


1.50 
.75 
-0- 


By  Summing  the  scores  on  all  factors,  the* prime  -Sponsbr  had 
^  •  -» 

*  **  .  * 

a  straightforward  methbS  for  comparing  relative  Effectiveness 

amo^xg  components  .or  contractors  v^ithin  components. 

? 


/ 
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*      3*   Developing  a  Composite  Index  ^ 
Composite  indexes  have  a  mamber  of  distinct  advantages 


as  a  means  for  facilitating  the  assessment  of  effectiveness. 
These'  include  the  facts  that  they: 

are  more  thorough  and"  equitable  ;than 
less  inclusive  systems 

/ 

#       present  data  in  a  simple,  concise  for:m 

*  V  •       place  different  programs  on*  a  common 
basis  for  -evaluation  purposes 


i 


T 


aid  in  the  establishment  of  shared 
evaluation  criteria*  for  each  project 

> 

help  to  addtan  element  of  direction  . to 
the  establishment  of  your,  evaluation 

Plan 

foster  healthy  competition  between  con- 
tractors and  components 

identify  those  performance  criteria 
which  your  PIC  Considers  most , critical . 


However,  as  the  two  foregoing  examples  demonstrate, 
there  are  a  variety  of  factors  which  can  go  into  a  com- 
posite index i  and  any  number  of  ways  for"  weighting  those 
factors.     If  you  decid£  that  a  composite  index  might  be 
appropriate  and  useful  for  evaluating  your  PIC's  projects 
and  components,  you  will  have  to  determ^e  what  factors  tq> 
include  and  the  weigfct  to  assign  td  each  factor4*  '   As  you 
can  imagine,  this 'would,  be  no  easy  task. 

Even  with  its  comprehensiveness,  however,  the  composite 
_index  will  not* make  decisions  for  you.  The  composite  index, 
as  the  other  ef f ectiveness  indexes  discussed  under  this 
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heading,  is  just  anothQf  means,  for  trying  to.  make  your 
decision-making  more  informed,  easier,  and, hopefully ,  better; 
Finally,  the  foregoing  discussion  has  focused  primarily 
_upon  the  use  of  'indexes  as  they  relate  to  the  participant 
side  of  your  program.     Obviously,,  the  same  principles  and 
techniques  can  be  applied*  to  the  EGS  side*.     The  essential  4 
consideration  in  doing  this  is  to  select  common  indicators^  and 
measurement  criteria  which  have  relevance  for  all  of  those 
projects  which  you  want  to  compare*. 


'  C.  -  DOING  IMPACT  EVALUATION-     m  * 

The  last  method  to  be  examined  under  this  step  is 
impact  evaluation.     As  noted  in  the  descriptions  of  this 
step,' impact  evaluation  is  an  attempt "to  measure  the  extent 
to* which  your  PIC 's  program  has  brought  about  change* and/or 
accomplished'  its  broader  programmatic  gqsls.     Ther^  are 
three  major  areas  for.  doing  impact 4  evaluation : ' 

•  participant*  Change    ;  / 
,  •    \  Cost-Benefits  - 

•  •  Community  or m Institutional  Change v 

s        This  'section  provides  only,  a  brief'  introduction  tg  each 
of  th^se  typek.     Doing  any  fojrm  of  impact  evaluation .  would^ 
require  that  your  PIC* secure  additional  data  and/pr  engage  \ 
in  substantially  more  advanced  data  analysis  *  than  was  dis- 
cussed  up  to  t*his  point.     Therefore,  if  your«PIC  decides' to 
undertake  an  evaluation  of (a**-4mpact  nature,  you  should 
ensure  that  you  alseafdy  havk  or  have  access  to  the  profes- 
sional assistance  necessary  to  get  the  best  results  possible 
from  such  an  undertaking.  ,  _  * 

*V    .  -   '  ■  3S3   •    •  ■ 


Participant  Change  #_  I 

Impact  evaluations  on  participants  normally  lopk^at 
the  services  (treatment)^  provided  by  a  program  to  ascettai 
what  the  actual  effects  which  can  be  attributed  to  the  % 
program  are.     T^p^Lcal  measures  whi.ch  are  used  to  look  at 
impact  in  terms  of  participants  include: 

•  earnings  gained 

•  •   private  sector  jobs  secured 

•  promotions  obtained  for  OJT  participants 

w        new  private  sector^  jobs  for  retrained  par- 
ticipants. * 


J 


\ 


1 .     Participant  Impact  Evaluation  Process 

A  standard  impact  evaluation  ]?rocess  for  participants 
wtyuld  include  the  following  elements: 

J  '  a.*  The  target  group,  of  persons  for  a 

specific  program^and  the  goals  of 
the  program  in  /terms  of  changes  ^ 
desired  are  identified. 

V 

b.    A  study  sample  is  drawn  from  the  9 
parent  population  composed  of  the 
♦    entire  potential  target  group  ffor        .  , 
a  given  program. 

e.    The  sample  is.  tandomly  assigned  to 

'   either  a  treatment  group  or  a  w 
"control"  group.  (Alternatively, 
instead  of  random  assignments  to  a  / 
1   tr^atmefit  group  and  a  "control" 

group,  a  "comparison" .group  can  be 
# developed  of  persons  as  similar  as 
possible  to  the  treatment  group  and 
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their- experience  tracked  to  pro-.  * 
vide  the  .peasure  of  what  would 
have  happened  in  the  absence  of 
the  prograip.l 

A ^re-program  measure  is  taken  v 
for  both  groups  of  indicators  and 
characteristics  which  the  program  > 
intervention  is  to  change. 

The  treatment*  group  is  exposed  to 
the  program  while  the  control  group 
(or  comparison  group)  receives 
either  tio'  service,  the  standard 
service  (not  including  the  treat- 
ment) ,  or  a  placebo  service. 


'At  the'conslusion  of  the  treatment, 
one  or  mote  post-measures  are*  taken 
of  both  groups  on  the : indicators  and 
characteristics  previously  examined. 
Tl/e  interval  between ,  completion  of 
the  progtam  and  the  post-tests  is 
determined  by  an  estimate  of  time 
needed  for  a  stable  condition  to 
developx 

The  pre-measurement  to  post- 
measutement* changes  ^in  the  indi- 
cators are  calculated  ,for  each, 
group ,  and  the  changes  experienced 
by  tfre,  control  group  (or  comparison 
group.)  netted  from  the  treatment 
group. 

The  result  is  then  interpreted  as 
the  effect  of  the  program.  The 
net  change  is  attributed  to  the 
program,  and  the.  problem  ;Ls  said 
to  have  been  reduced  by  the  amount 
of  the  net  change.  ' 
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Two  other  Useffll  alternatives  for  looking  at  participant 
change  which  do  not  require,  the  use  of  'either  comparison  or 
control  groups,  as  discussed^in  the  example  process  above , 
are:  *   \  ,  ^  * 

•  time  series  design 

•  follow-up  studies. 


r 


eries  Design 


j\  extension  Mf  before-after 


7 


.The  time  series  dpe-1 

.comparison  which,  despitrs^B^Ka'tn  limitations,  can  yield 

■  p 

useful  information  on  proaffaflT effects .     The  design  estimates 
the  net  effect  of *the  program  by  measuring  against  an  ' 
extrapolation  of  the  p5st  expedience  of  program  enrollfces. 
Tc*  do  this,  the  design  calls  for  a  series  of  pre-program 
measurements  to  establish  the  participants'  preprogram 
status.     After  the  program,  a  series  of  post^program wnfeasure- 
ments,  are  taken.     Every  possible  consideration  —  such  as 
the  tendency  of  employment  and  earnings\trends  to  deteriorate  - 
just  befoffe  program  enrollment,   the  state  of  economic  activity, 
the*  nature  of  wag§  trends',  and  so  forth  — *  are  examined  and  *  ■ 
then  judgments -4nade  as  to  what  would  have  occurred  in  the 
absence  of  the  program.     The  post-prograntt  measurements  are . 
theri  compared  to  the  adjusted  and  extrapo^ted  trend  of  the 
pre-program  measurements  to^ yield  an  estimate  of  the  effect 
of  the  progranv.  \  m 
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3.     Follow-Up  Studies  '  % 

Follow-up  studies  are  undertaken  for  basically  three 
purposes: 

Client  Assistance:  to  determine  the  immediate 
^success  or  immediate  need  for  additional  ser- 
9  %         vices  by  a  ciient  who  has  jfec^ntly  left  the 

program.    Thi£  short-term  follow-u^^nerally  1 
is  undertaken  thirty,  sixty,  or  ^inefs?  days.^  . 
-  after  a  client  leaves  the  prografh.* 

*  •  #  short-term  Assessment:     to  get  some  judgmental  • 
insights  into  what  £hat  longer-term  evalua-  • 
tion  'is  likely  to  show,  and  to  begin  to 
identify  program  weaknesses  a*nd  make  changes 
while  services  Are  still  ongoing.     This  is„ 
also  a  sho£t-term  folldw-up  method. 

Often  telephone  calls  to  a  limited  runrfber  of 
£  recent  terminees ,  j>a,sed  on  who  is  available 

rather  than  randomness  and  done  without  reliance 
on  jcontrolled  comparisons,-  can  provide  useful 
insights  at  low  cSst,  so  long  as  the  evaj-uatx)? 
keeps  in  mind  that  this  is  only  a  shorty ternf 
*  *  indicator  for  primarily  managerial  purposes 
and  cannot  replace  longer-term  and  more  care- 
>  ful  evaluation.' 

.     •  LOngTterm  Evaluation:     to  determine  the  longer- 
term  success  of  the  program  .graduate.  This 
fbllow-up  is  conducted  ninety  days  .to  a  year 
'*  ^or'more  after  the  client  has  left  the.pro-    .  , 
gram  to  dollect  information  on  what  the  client 
--has  gained  "tfrom  participation,  in  t^rms  of    ^  • 
'    wages,  .job  stability,  and; so  on.     Th^  longer-.  \ 
term  follow-up  might  include  dropouts 4  and 
*  *  nonplacements  as  well  as  placements  for  . 

purposes  of  comparison.  . 

Anyone,  or  all  of  these  forms  of  follow-up  may  be      *  { 
useful  to  "your  PIC  -  not  only  on  the" participant  side,  but  also 
in  looking  aV  the  Continued  impact;  of  EGS  'services  * 

*  '        ■  .   '  •      \  \  y  • 

Cost-Benefits  Analysis 

'Another  way  of  looking  at  the  impact  of  your.PIC's  pro- 
grams is  in  investment  terms  of  costs  and  benefits   (or  return 


y 


• 
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on  the  investment).     Cost-bqnefit  analysis  is  an  attempt  to 
assess  all  of  the  costs  and  benefits  of  a  project  ot  program 
in  equivalent  terms.     There  are,  a  variety. of  cost-benefit 
approaches  which  have  been  Revised. 

The  chief  drawbacks  to  thes£  approaches  are  that  (1)  it 
is,  very  difficult  to  define#what  should  £e  included  or  cal- 
culated as  either  |  cost  or  benefit,  and  (2)  even  after  working 
definitions  are  operational ize<S,  a  large. amount  of  estimation 


Only,  that  you  should  be  aware  of  its  limitations  and  its  . 
actual  explanatory  value. 

■ 

,  1  •     Types  oS.  Cost-Benefit  Analysis  . 
These  include: 

•  ftet  present  value  approach:     subtracts  the 
present  value  of  costs  from  the  present 
vflue  of  benefits.     If  the  result  is  posi- 
tive, the  program  is  viewed  as  worthwhile. 

•  \  benefit-cost  ratio;     uses  a  ratio  of  the 

present  value  of  benefits  divided  by  the 
present  value  of  costs.     If  the  ratio  is 
greater  than  1,  the  program  is  viewed 
as  t«>rthwhile. 

•  internal  rate  of  return  method:  computes 
the  discount  rate  required  to  equate  the  . , 
present  value  of  the  future  stream  of 

N  *        £enef its  to  the  present  value  of  costs. 

If  the  resulting  interest  rate  is  greater 
than  the  "opportunity  costs"  of '  the  funds 
used  in  the  program,  the  investment  ijr*< 
worthwhile., 

payback  method:    divides  the  present  value 
of  costs  by  the  pre-  to  pos^^program  in- 
crease in  ,annual  trainee  taxes.     The  re-  * 
suit  is  the  number  of  years  it  would  take 
for  the  taxes  paid  to  equal  the  program 
costs.     A  variatioft-of  the%payback  method 
is  the  "earpback"  method  which  divides  the 
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present  value  of ^costs-by  the  average  pre- 
to  post-program  net  cjain  of  annual  income 
by  participants.     The  result  is  a  figure- 
which  represents  the  number  of  years  re- 
quired for  participant  income  increase 
to  equal  program  costs.  ^ 

The  .figure  which  follows  prc>vitles  a  simple  comparison  of 
these  four  basic  approaches:  - 


Cost  Benefit  Approaches 


Net,  present  value 

,  Present' Value\of  costs  •  *  $1,000 

Present  value  of  benefits  >      i, SQO 

.  f        Net  present  value     •  #  $  500 

1         «      •  °  \ 


Benefit-cost, ratio  % 

Present  value  of  benefits  *      '  $1,500 

Present  value  of  costs  ^  1,400 

Internal- r^t£  p,f*rgturn 

Present  value  of  benefits  .  ^  $1,500 

Present  value  of  costs  *  n         "     1,000  _j   500 


=  1.5 


4Inccme»  of  .  .  •  !  '    fc$  500 


1,000  =  °-50 


Payback 

Present  value' of  qcfstU  r  $1,000 

•  Pre-  to  pe^trprogram  increase      ;   *  ~  * 

*        in  trainee  taxes        *  , 


200  =  five  yeirs 


* 
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2.     Applying  a  Payback  Method 

The  payback  method  is  the  least  complex  and  easiest 
^form  of  cost-benefit  analysis  to  apply,    A  PIC  devised  the 
following  formula  to  determine  the  required  payback  period 
to  the  community  for  the  classroom  training  projects  which 


'  it,  sponsored:  ± 


Payback  in  mos._- 


Training  Cost 


(Post  CETA  Wage  -Jx  Nq    Ent^  ^^^^  x  ^g    Mo^  Working  Hifs. 


%  \;  Pre  CETA  Wage 

rt  applied  this  formula  to  its  four  classroom* training  projects 
and  obtained  this  ranking:.  p/  ' 


Project 
A 

•  B. 
C 
D 


Payback  Period  in  Months  • 


1.8 
7.0 
7.2 
20.2 


This  ranking-  is  especially  interesting  when  reviewed  in 
conjunction  with. the  same  projects'  'results  in  terms  tff  place- 
mentvrates  as  presented  below: 


Project 


Payback  Period  Placement  Rate 


A 
B 
C 
D 


1.8 
7.0 
7:2 
20.2 


"48.1%  ' 
84. 8*** 

54.5% 

'83.7%- 


These  comparisons  dramatically  illustrate  the  pointy  that  youj, 
.PIC  should  not  depend  solely  upon- single  measures  or  indicators, 
to  assess  the  impact  of  your - pro jects,  or  components. 

Community  and  Institutional  Change 

Other,  possible  areas  which  could  be 'examined  and  qu'es^ 
tions  which  might  be  asked  in  terms  of  broader  impact/  especially 
•as.  they  relate  to  your  ?I&8  ^EGS  activities,  include: 

i 
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Institutional  Impact  of  Title  VII    *  ' 

•  Has  the. PIC  stimulated  new\busiriess  participa- 
tion in  local  employment  and  training  efforts? 

•  Has  the^PIC  stimulated  new  responsiveness 
on  the7  part  of  "traditional"  empldymenj;  and 
training  deliverers  to  the  needs  of  business? 
13  there  a -comprehensive  planning  approach 
between  PIC  and  the  local  Employment  and 
Training  Council? 

•  Has  the  PIC  stimulated  any  areawide  plannijjflg 
between  different • Prime  Sponsorship  areas 
and  other  p\[Cs? 

%Attitudinal  Impact  of  Title  VI I v 

•  How  has  the  PJC  program  affected  the  attitudes 
of  employers  toward  the  program,  toward  employ- 
ment and  training  programs, in  general,  and 
toward  eligible  clients  as  potential  employees? 

•  How  has  the  PIC  program  Impacted  on  the  atti- 
.  tudes  and  motivations  of  participants? 

.  ^Patterns  of  Service    *  *  .  * 

■"•What  are  the  characteristics  of  the  partici- 
pants being  served?    How  do  they  compare  to^ 
regular  CETA  clients,  goals,  and  some  state- 
ment of  "universe  of  need"? 
»  What  ar6  the  characteristics  of  the  businesses 

being  S'erved?    Where  do  they  stand  #in  the 
fc  overall  economic  structure  of ^the  area?    *  . 

»  » 

Cost . 

0  What  reasonable' cost  calculations  can  be  made 

*  .      about  somd  of  the  above  impacts? 

"Stage  H.     Evaluate  Results,"  pp. III-355  -  363  describes 

*  *  .  . 

techniques  that. you  might  use  to  gather  and  analyze  impact 

*  x,  » 

data  of  the  institutional  or  community  change  type. 


CONCLUSION 

In  conclusion,  as  the  prece^ir^r  discussion-  indicates, 
there  are  numerous  evaluation  opfej^ns  available  to  your  PIC. 
You  should  pick  -those  evaluation  approaches  and  measures  whic 
yrfu  feel  will  give  you 'that  information  whidh  you  need  to 
know  in  such  a  way  that  it  can  assist  you  in  making  ^ecisions 
and -furnishing  direction  to  the  PICv 

^  s  '  111^279   331  ^ 


Mayor's  Office  of  Manpower,  City  of  Chicago, 
Planning  and  Evaluation  under  CETA* (Mash-  ' 
mgton,  D.C.;  U.S.  Department  of  LalSDiTf 
January  1976).  ^  w 
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Step  2 


Assign  Evaluation  Responsibilities 

ar\d  Establish  TrocedureF  / 


The  purpose  of  this  step  is  to  establish  the  general  frame- 
work for  administration  of  your  PIC1 s -evaluation.  This 
involves  considering  the  Prime  Sponsor's  evaluation  approach; 
selecting  "staff  to  administer  the  evaluation;  establishing 
the  process*  for' a  systematic  approach  to  th§  evaluation; 
and  determining  the  fQle  of  tfid  Council,  % 


•  Evaluation  Review  Process;     basic  method  and  propedures 
to  be  adhered  to,  in  assessing  PIC  performance,. 

•  Evaluation  Responsibilities:  roles  and  duties  foA  a. 
individuals  to  be  involved  in  the  conduct  of  the  \  ' 
evaluation. 


11 


A 


c 


\ 


 «  • 

fl(  1(    Know1*qe/Cap»bimies  Rattng  . 

Review  the  information  provided  above.  Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  this  step  on  the  scales  below. 


ERLC 


• 'Knowledge 
* 

•  Capabilities 


\  - 

1  - 

— : »  » 

i 

-1 

Excel-  ' 

Above* 

^Lver- 

Below  * 

Poor 

lent  " 

Avg . 

w  age 

Avg. 

1 

 1  

— ^—J 

Excel- 

Above 

Aver- 

Below 

f  Poor 

lent 

•  Avg. 

age  * 

.  Avg .  « 

»  * 

Based  upon  yowr  ratings,  decide  whether ^you  need  to  read  the  content 
information  on'this  step  wfyich  follows  immediately  or  proceed  to  the 
next  step.  •  • 
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PME  PROCESS .NOTES/ACTION- PLANNING  >AGE 


'NOTES   (Points  that  are  important  which  you  want  to  remember) 


QUESTIONS   (Items  on  which  you  are  ^unclear  or  want  to  get  additional 
infprmation) 


\ 


ACTIONS   (Specific  steps  that -you  intend  to  take  to  -implement 
PIC's  PME  process  in  this  .area)  '  \ 


your 


Steps 


Reso<yrces/Persons 


•  Time  Prame 


er|c> 
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INTRODUCTION* 


Once  vqu  have^decided  on  the  scope  and  nature  of  your 
PIC1 so evaluation,  the  next  step  is  to  identify  ^he  basic 
tasks  to  be  performed  c^nd  assign  responsibility  for  their  * 
performance.     This  involves: 

establishing  procedures  for  an 
.  •   organized,  systematic  approach  to  / 
the  evaluation 

•       determining  the  role  of  the  Council 

«   •  *     selecting  sta'ifF^to  administer  the 
.  .  •  evaluation. 


As  in  the  development  of  your  management  or  monitoring 
plan,  a  UMic  consideration  here  must  be  the"  quality  and 
scape  of  ttie  Prime  Sponsor '  s  ^valuation  system^.     If  that^ 
-•system*  is  adequate,  you  may  want  to  interface  with  it  and 
rely  upon  it  to'  collecjt  and 'array  the  data,  and  generate 
the  reports  that  you  need  for  evaluation  purposes. 

This  will  reduce  the  level  of  staff  effort  that  you 
have  to  put  into  the  evaluation  process.-   If,  on  the  other 
*r  hand,  you  find  the  Prime  Sponsor system  is  inadequate 

for  yo\ir  PIC's  needs,  then  you -will  have  to  expand  your 
#  responsibilities  accordingly. 


Evaluation  Review  Process 

The  f  irs.t  task  which  your  PIC  should'  accomplish  within 
this  step  is  to  establish  the -basic  operating  procedures 
and  working  Arrangements  for  administration  of  your  \ 
evaluation  efforts. 
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"  .  ;    A.     ESTABLISHING  THE  REVALUATION  REVIEW  PROCESS 

''*■»■  ■  •     •  , 

*k  Your  PIC's-  evaluation  review ; process,  is  the  method  and 

a^^rpach  it  wilJLv  employ  to*  gQvern  and  direct  its  evaluation 
activities.     Th'e  Joey  issue  here  is  to*  establish  the  basic 
*  operating  procedures  and* working  arrangement^  for  administra- 
tion of  your  evaluation  efforts.     These  items  should  be  speci-  „ 
9x   . f ied  at  the  outset  of  your  evaluation  planaing.     General  ques- 
#        *:ionsv  tp  be  answered  inqlu.de:  .  v 

%  ,    Who  .will  .have  ultimate 'responsibility 
*  "for  administering  th6  evaluation  pro- 

~V  gram?  .  » 


Who  will  stipervise  the  ©valuator  (s)  ? 

What  prdcedure^wiir  be  devised  fQr  the 
systematic  review  of  evaluation  ^Lans, 
methodologies,  data,  and  findings? 

0ho  will  be  involved  in.  the  analysis  of 
evaluation  reports  cfnd  in  the  inter-  . 
pretatiojp  $f  f^pdings? 

What  steps  will  be  instituted  to  ensure 
that  deadlines  are  observed? 


More  specific  issues  to  be  resolvedrarej 

•  the  overall  purpose  of  your  PIC's  evaL- 
, nation;  ♦ 

***  :  - 

expectations  regarding  the  evaluation  - 
/  what  it. should  enable  you  to  do^ 

•  the  resources  which  will  be  expended  v 
for  the  evaluation  * 

•  •  -   AethodS  fqr,  coordinating  evaluation 

activities  with  those  being  undertaken 
"by  the  Prime  Sponsor 


/ 


•  >  » 

due  dates  fQr  any  reports  or  docu-r  *  , 

ments  from  the  evaluation 

the- methods  for,  monitoring  tHe  eval- 
uation {e.g.,  monthly  meetings  or 
progress  reports) 

responsibilities  of  .staff  in  the  * 
evaluation  r 

tjie  persoA{s)  to  whom  staff  charged  / 
with  evaluation  responsibilities  should 
report  and  be. accountable 

policies  regarding  the.  retention 'of 
records  and  raw  data  ft 

procedures  for  resolving  problem  issues 
as  they,  occur 

the  role  of  the  Council  in  the  evaluation. 


B.     DEFIN^G  THE  COUNCIL*  S  ROLE 


Defining  the  role  of,  the  Council  ih4 the  evaluation  i§ 
especially  important  task.  Regardless  of  the  specific 
responsibilities1  that  you  dec^fln^oj^,  it  is  desirable 
that  the  Council  play  an  actr^PKle  in  the  administration 

of  k  evaluation  activities.     Possible  options  include: 

*  " '     V     "      "  *  • 

#  appointing  a  subcommittee  of  the 
Council  to  review  and  monitor  all 
evaluation  activities  s 

k  / 

•  having  the  evaluator(s)  report  to 
the  Director,  who  periodically 
apprises  the  Council  regarding  the  * 
evaluation  program  and  its  progress* 


.j 

is  an 


Evaluation  Subcommittee 


9 
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.f-'If  you  decided  toy appoint  a  subcommittee wto  -oversea 
all  evaluation -activities,  that  subcommittee  could 

•  participate  in  the  selection  of  an 
#    evaluation  staff  person  » 

•  *  designate  projects  to  be  evaluated 

— «.*  • 
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•  ensure  the  provision  of  adequate  re-  ^ 
sources  far  the  evaluation 

•  review  and  critique  the  evaluation 
design 

•  monitor  progress  on  the  evaluation 

•  receive  evaluation  reports  and  findings, 
and  comment  on\them  before  forwarding 

to  full  Council 
*  •  * 

•  make  recommendations  to  the  full  Coun- 
cil  on  future  ^jj^j^J  tjfeS  based  on 
evaluation  f  ir  * 


If,  as  part  of  your  ntfhitQi;in£  planning,  you  decided  to 
establish  a  Subcommittee ^pp  monitor,  program  performance,  it 
might  be/*advisable  to  combine  the' monitoring  ami  evaluation 
functions  into  a  single  subcommittee  for  program  r£vi6w  and 
evaluation. 

°  To  repeat,  however/  no  matter,  how  yqur  PIC  defines  its 
exact  role  and  responsibilities  in  the  evaluation  area,  it  is 
esfiential  that  the^i^ncil  *be  involved  to  a  meaningful  extent. 

C .  *  SELECTING  EVALUATION  PERSONNEL 

In  order  to  determine  who  should  be  responsible  for  the 
actual  conduct  of  your  PIC's  evaluation,  you  should  answer  the 
following  questions: 

|  .   •  How  muofi  emphasis  is  being  placed  on 

tjie  evaluation  at  your  PIC? 

<  t 

•  What  resources  will  be  made  available 
„  to  implement  your  evaluation  rplan? 

•  6o  anycurrent  stafff  members  have  the 
capability  (or  the  time),  to  perform 

*  evaluation  research?  \ 

•  Would  it  make  more  sense  to  hire  a  7 
full-time  evaluation  specialist  or 
employ  private  contractors  on  an  ^ 

ad  hoc  basis?  * 

The  central  issue  in  the  selection  of  evaluation  personnel 
is  the  amount  of  eiftphasis  that  your  PIC  is  placing  on  evaluation,  ' 
and  the  resources  ^that  will  be  made  available  for  its  implementation, 
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?  ■.  ^  : 

;  .  -      ,  ■     r  . 
*  ...        %    ■.  **  . 

Limited  Scope  Evaluation   .  ^  J 

If  your  PIC  has  a  ve^y  narrow  focus  or 'limited  scope 
for  its  evaluation — one  Which  involves  using  existing  data 
sources — then  it  can  probably^ J^e  accomplished  with  incumbent 
staff.;    In  order  to  meet  basifc  PIC  evaluation  requirements, 
a  staff  member x n$ed  not  have-a  wealth  of  experience  or 
formal  education^,     It  is  more-  important  that  the  individual  (s) 
selected  to  perform  evaluations  be  familiar  with  pic  programs, 
operations,  and  clientele,    .The.  essential  requisites  for 
basic  evaluation  research ^are  4  logical  mind,  attention  to 
*f  detail,  arud  the  ability  to  follow  instructions. 

The  primary  requirement  for  an  effective  evaluation 


ERIC 


using  in-hous^  or  existing  perflfcanel*  is  that  a  person  or 
persons  be  given  definite  responsibility  and  specific 
assignments  for  the  evaluation.     Studies  have  also*  fout)d 
that  when  that  person  is  a  "specialist",  i.e.,, spends  most 
of  his/her  time  on  monitoring  and/or  evaluation  activi- 
ties, as'oppdsed  to  a  "generalist%  xi . e.  ,  must  get 
involved  in  myriad  other  activities,  evaluation  tends  to 
be  accomplished  in  a  more  cotnplete,  professional,  and 
utilitarian"  mariner  I  _  ^ 

•  $ 

Broader  Scope  Evaluation 

If,  on  ,the  other  hand,  your  PlC  decides  to  undertake 
a  broader  scope  Of  more  sophisticated  evaluation — one  which 
requires  obtaini/ig  additional  data  using  techniques  such 
as  survey  research  or  foll6w-up  jstudies--then  you  may  want 
to* hire  an  evaluation  specialist  or  an  outside  contractor, 
tf  you  must  go  outside  to  -secure  evaj^ation  assistance, 

there  are  three  sets  of  skills  that  yoto  should  look  for: 

■*  *  » 

•  Conceptual  Skills^    ability'  toh see 
the  entirely  of  ybur  PIC's  program 
^activity^  to  translate  >goals  /into  mean- 
ingful indicators  ^nd  measure?  which  can 
be  used  to.  assess  progress  toward  those 
goals.       *  v 
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•  Methodological  Skills;    ability  in  hreas  such  ' 
-  as  revaluation  design,  statistics,  econometrics, 

quantitative  methods*  and  questionnaire  and  * 
s\irvey  research  construction. 

•  Communication  Skills':    ability  to  interact  . 
well  with  others  both  in  obtaining  data«and 

*       presenting /Cv^luatj.pn  findings. 

All  three  of  the  sfkills  are  equally  ipportanfc  to  ensuring 
rigorous  and  useful  evaluations  for  your  PIC. 


Managing  the  Evaluation  Process 

'Regardless  of  how  you  s^taff  your  evaluation  effort,  you 
should  create  a  definite  work  plan  for  monitoring  staff  per- 
formance  and  progress.    Review  sessions  should  be  scheduled 
between  the  evaluation  administrator  and  staff  to  confirm 
that  the  assignment  is  proceeding  in  a  timely  and  efficient 
manner.     Close  'and  systematic  monitoring  will  ensure  that  any" 
problem  issues  are  addressed  before  th^  jeopardize  the  eval- 
uation." 


CONCLU 


)  . 

In  conclusiprf,  onoe  you  have  established  a  basic  evalua- 
tion review  process  and  assigned  responsibilities  and  sfaff 
related  tQ  that  process,  you  have  created  the  fr.amework' within 
which  to  accomplish  your  PIC's  program  evaluation. 


Manpower  Administration  (now  Employment  and 
Training  Administration   [ETA]),  Program 
Assessment  Guide   (Washington,  D.C. :  U.S. 
Department  .of.  Labor,  April,  1974)  . 


Mayor's  Office  bf  Manpower,  City  of  Chicago, 
Planning  and  Evaluation  under  CETA  (Washing- 
ton,  D.C. :  U.S.  Department  of  La bos ,  Jan.  197  6 


Ohio  State  University^  A  Formative  Evaluation  of  the  Private 
Sector  Initiative  Program  (Columbus,  Ohio:  Ohio  State  Univer- 
sity, 1981).  r~~~~ 

*  - 
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Construct  Evaluation  Research  Design  (Plan) 


The  purpose  of  this  step  is  to  construct  your  PlC's  evaluation 
reseatch  design.     An  evaluation  research ^design  is  a  detailed 
plan  for  implementing  your  evaluation.     The  design  specifies: 
the  purpose  of  the  evaluation;   its  objectives;  data  sources; 
methodology — techniques  for  data  collection/  aggregation ,  and* 
analysis;   instruments;,  worj^  plan  , and  schedule;  and 'reports  to 
be  produced  as  a  result  of  the  evaluation. 


•  Evaluation  Research  Design 
evalu4t ion . 


detailed  plaji  for  implementing 


•  Hypothesis  -  proposition*,  that  is  Consistent 9 with  known  facts 
and  therefore  presumed  as  'possibly  true,  but  which  requires 
further  investigation  for  verification. 

•  Validity  -  extent  to  which  evaluation  actually^neas*ires  what 
it  is  intended 'to  measure  taking  all  important  factors  into 
account, 

•  Reliability  -  extent  to  v^ich  evaluation  would -produce  tiie 
same  or  consistent  results  thro'ugh  repeated  tests  over  tims 
using  identical  procedures  and  iilstrufnents. 


M(  i(   Knowledge/Capabilities  Rating" 

Review  the  information  provided  above.  Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  this  step  on*  the  scales  below. 


•  Knowledge 
i 

•  Capabilit 


ielk 


L 


Excel- 
-  ljpnt 


Excel- 
lent 


Above 
Avg, 


'  Aver- 
*  age 


Above 
Avg. 


Aver- 
age 


Beiow 
Avg. 

_L 


Below 
Avg. 


Poor 


J 


Poor 


Based  upon  your  ratings,  decide  whether  you  need -to  read*  the  pontent 
information  oh  this  step  which  ^follows  immediately  or  proceed  to  the 
next  step. 


9 
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PME  PROCESS  NOTES/ACTION  PLANING  .PAGE 


nolrES^PoirT^c^t  are  important  which  you  waffc  to  remember) 


QUESTION^^Ite^  you  are  unclear  or  rwanf  to  get  additional 

"         •      '    information)  % 


ACTIOM^^SpeciTi^^tep^tha^y^  intend  to  take  to  implement  your. 
^felC's  PME  process  in  this  area) 

Resources/Persons  .  Time  Frame 


Steps 


-1  • 


■er|c 
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v  INTRODUCTION 

f 

'"'       '       .    -  ^      .  > 

*    The  next  step  after  determining  the  scope  and  nature 

y 

of  your  evaluation  is  to  construct  your  PIC's  evaluation 
research  design.     An  evaluation  research  desigp  is* a  formal 

.written  statement  of  the' detailed  plan  for  implementing  your 

< 

PIC's  evaluation.  j  * 


A.     CONDUCTING  AN  EFFECTIVE  EVALUATION  * 

-A 

Regardless  of  the  type  of  ^valuation  your  PIC  decides 
to  undertake  or  the  nature  of  your  plan,  there  ^e  three 
general  concepts  which  are  central  to  the  conduct  of  an' 
Affective  .evaluation.     These  concepts  are: 

m       Hypotheses  ,  ,  * 

•  •       Validity  .  *  , 

m  Reliability 

Testing  Hypotheses  *  t 

All  evaluation  research  begins- with  a  set  of  assumptipns 
about  the  program  gr  phenomenon  to  be  studied*     These  assump- 
tions m^y  be  formally  expressed  in  hypothetical  terms,  or 
they  may  be  stated  only  -indirectly.    4*  is  not  always  neces- 
sary for  your  PIC  to  cteyelop  written  hypotheses  for  its  * 
evaluation*  but  they  are  useful  in  that*\they  serve  to  focus 
thfe  study  £>r\  the  key  issues  to  be  examined.  /  % 

Technically,  a  hypothesis  is,  a  proposition  that  is  con- 
sistent with  known  facta  and  therefore' presumed  as  possibly 

♦ 

%true,  but  which  requires  further  investigation  for  verifica- 
tion.   An  hypothesis  pust  meetx  the  thrae  criteria: 

•       It  must  be  apparent  which  of  its  con- 

•    cepts  or  elements  is  caused  and,  which  ; 
is  causal.  . 
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•        It  must  be  stated  in  such  a  way  that 

v  it  can  -be  tested.    ^  s/  *  ✓ 

i  /  %  ' 

m        It  should  express  its  concepts  in  both 
a  predictive  and  explanatory  manner. 

In  addition,  ai*  hypothesis  must  be  stated  in  precise,  logical, 
*  and  positive  terms,  as  the  examples  below  indicate: 


HYPOTHESBS 


EXAMPLES 


Unemployment  will  be  less  among  those  teenagers  who 
successfully  complete  a  vocational  skills  training 
program  than  among  nonpar tipipants. 


The  average  hourly  wage  will  be  higher  for  those', 
target  group  members  who  participate  in  and  com- 
plete classroom  training  than  for  those  eligible 
participants  -who  do  not  participate  *in  AacH  a  pro- 
gram. ^ 


More  jobs  will  be  created  for  CETA-el igible 'par- 
ticipants through  «an  economic  development  linkage 
Project  than  if  no  such  project  existed. 

J  /      *   ,  

 1-  z  r^~~ 


The  results  of  the  research  arc  in  effect,  the  test  of 
the  hypotheses     The  hypotheses  are  either  confirmed  or  in- 
validated t>y  your  res'eafch  findings. 

Hypotheses  help  you  to  focus  your  evaluation  a\nd  to  obtain 
answers  to  those  questions  which  you  deem  most  essential.  By 
^nsuring  the  validity  *and- reliability  of  -your  research  effort, 
you  can  be  certain  that  those  answers  whioh  you  obtain  are  correct. 

Ensuring  Validity  "•  '^^J^tf 

In  or8er  for  your  evaluation  design  to  be  moa?  meaningful 

and  explanatory,  it  must  possess  internal  Validi^E  '  If  a  design 

has  internal  validity,  it  has  the  capacity  to  distinguish 

between  the  effects  attributable  to  your  program,  and  those 

resulting  from  other  factors.  * 

^or  example,  an  evaluation'of  a  training  program  that 

compared  the  post-training  income  of ^trainees  and  nontrainees 

would  have  validity  only  if  all  other  factors  affecting  income^ 

(e.g.,  attendance,  gender,  type  of  employment,  etc.)  *ere  > 

isolated.^  ,  v  ' 

t. 
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Your  research. design  should. enable  you  to 

•  identify  external  variables  which 
Mnay  affect  program  results; 

.  > 

•  measure  the  extent,  of  their  inf lu-, 
enfce,  if  any;  and  / 

•  document  the  degree  to  which  results 
were' attributable  only  to^the  program 
in  question.  ^ 


Ensuring  Reliability 

If  the  findings  from  your  PIC's  evaluation  are  to  be  - 
most  useful  to  ydti  for  decision-making,  they  must  not  only 
fc€}  valid  (i.e.-,  measure^  what  they  say  b^ey  are  going  to 
measure  -taking  all  important  factors  into  account) ,  they 
mUst  also  be  reliable  (i.e. ,  produce  the  same  or  consistent 
results,  through  repeated  tests  using  identical  procedures 
and  instruments) . 

ToSfensure  reliability,  your  PIC  should  develop  a  . 
written  statement  of  the  specifications  governing  its  study. 
Each  methodological  step  must  be  described  in  detail,  as 
must  the  characteristics  of  the  individuals  or  entities 
who  axe  to  be  the  subject  of  the  evaluation.     This  will 
provide  you  with' a  blueprint  in  a  rigorous  and  reliable  . 
manner  which  caii  be  used  to  administer  your  e^lnation.  and 
give  you  the  confidence  that  your  findings  are  accurate  and 
adequate  enough  to  allow  you  to  take  those  steps . necessary 


foj:  program  review  and  modification, 


B.     PREPARING  YOUR  PIC'S  EVALUATION  PLAN 

•   •  r 

To  prepare  your  PIC's  specific  'evaluation^psearch 
design,  so  that  it  tests  those  hypotheses  and  ensures 


reliability  anci  validity,  there  are  several  questions  which 
you*  need  ,to  answer .  " 

Evaluation  Questions  ' 
These  questions  include: 

•  Why  is  the  PIC's  program  being  evaluated? 

•  What  are  £he  primary  and  secondary  ob- 
jectives (standards)  of  the  program? 

•  Which  of  these  objectives,  (standards) 
are  most  easily  measurable? 

•  What  are  the  hypotheses  ycfu  want  to  test? 

•  What  resources .are  available  to  Evaluate 
the  program?  ' 

•  How  much  time  is  permitted  for  the  eval- 
uation? ^ 

•  What  data  is  needed?  r  *  t 

•  Where  does  this  data  exist?  ^ 

•  How  can  this  data  be  obtained?* 

•  How  will  this  data  be  analyzed? 

•  How  will  ^he  findings  be  used? 

*  < 

^•.What  format ^will  be  best  for  present irfg 
your  evaluation  findings? 

Your  answers  to  these  questions ^will  determine  the 
ultimate  structure  and'  format  of  your  evaluation.^/  Once  you 
have  made  these  determinations,  they  should  be  summarized  and 
presented  in  a  formal  written  plan  for  evaluation  of  your 
program.  * 

Evaluation  Plan  ^ 

Your  PIC's  evaluation  plan  should^consist  of  the  following 
^sections: 

Purpose:     states  the  reasons  why  the  study  is 
being  undertaken.     Incorporates  a  brief  descrip- 
tion of  the  program  and  its  goals.* 
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r  ftationale;  states  succinctly  reasons  for, 
and  purposes  of,  evaluation,  and  how  .study 
results  will  be  used. 

Objectives;     specifies  objectives  and  scop*e 
for  the  evaluation. 

Methodology;     the  heart  of  fche  research  de-  , 
sign,  states  exactly  how  the  evaluation  will 
be  performed.  Details: 

•  data  that*  will  be  needed 

*    *  techniques  of  data  collection 

#  means  of  data*  aggregation,  and  Analysis. 

,  Instruments;    describes  the  types  of  instrji-. 
ments  and  worksheets  needed  to  oollect  and 
treat  the  data  and  discusses  the  process  by 
which  they  will  be  developed. 

*- 

Schedule;     sets  out  a  timetable  to  govern  the 
.  progress  of  the  evaluation.     Highlights  all 

deadlines  and  due  dates. 

»    •  * 

Reports;     indicates  the  types  of  reports 
that  will  result  from  the  evaluation,  in- 
eluding  their  content,'  format,  and  sub- 
mission dates. 

•  #The  pages  which  follow  provide  some  additional  informa- 
tion regarding  each  of  the'  elements.  % 


( 

•  m  1.    Statement  of  Purpose 

This  section  tells  why  you  are  doing-  your  evaluation. 

Specificity  is  desirable,  since  the  rest  of  ^the  design  and, 

in  fact,  the  activities  of  your  evaluators  should  be  rele-.. 

vant  to  the  purpose,  or  purposes,  of  the  study.  -^Ex^mples 

of  statements  of  purpose  follow; 

•       To  compare  the  effectiveness  of 
»  different  components  with  the  same 
purpose.  %  \ 

m       To  furnish  data  relevant  to  a  deter- 
mination of  whether  a  particular 
progrant  should  bS  continu^ 


•    *   To  determine  the  extent  to  which  pro- 
gram type  and  mix  contribute  to  th$* 
q  -         *  achievement  of  the  PlC's  overall  goals. 
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To  determine  the  relative  cost-effec- 
tiveness of  a--  number  of  programs 
concerned  with  ra  particular  target 
group 

To  secure  information  which  would  le^d 
to  improvement  in  the  management  effi- 
ciency of  a  project 

To  determine  the  quality  of  manpower 
services  *gdven  to  a  target  area  or 
group 

To  assess  the  status  of  a  project  versus 
its  objectives,  and  the  causes  for 
failure  to  meet  objectives 

To  assess  whether  the  attitudes  of 
local  employers  regarding  CETA  have 
changed  as  a  result  of  PIC  programs 

TA  determine  whether  worJc-related  atti- 
tudes of  participants  have  changed *as 
the  results  of  participation  in  a  pro-, 
gram,  or  in  a  group  of  programs. 


2.     Rationawfe"^  .  , 

This  section  describes  the  uses  to  vAich  the  stu£y  will  « 
•be  put  and  the  reasons  for  .doing  it.     The  rationale  should 
state  why  the  e^reas  chosen  for  evaluation  can  provide  beneficial 
insights  for  planning  or  administration.     If  the  evaluation 
was  planned  as  part  of  yjur  progjam  planning  and  the  prograip 
Wag. ,jat:radtured  accordingly,  the  rationale  should  explain  this. 
The'  rationale  for  each  area  of  inquiry  should  be  ,as *  succinct 
and  clearly  stated  as  possible.     Abbreviated  examples  of     •  . 
rationales  follow: 

f      9       When  the  OJT  Project  was  funded  and  con- 
tracted, attitudi^al  questioning  of  the  em- 
ployees at  the  time  of  enrollment  and 
three  and  six  months  after  was  agreed  upon 
*  and  written  into  the  contract.     It  is  also 

.important  to  determine  whether  the  employer 
has  been  satisfied  by  the  work  of  the  par- 
ticipants.   Whether  the  program  is  con- 
.  tinued  wfll  depend  partly  on  the  evaluation. 


\  III-298 


40<J 


•  •     New  areas  of  employer,  nfeeds  have  become  . 
apparent  since  the  beginning  of  the  PIC 
program*    A  study  should  be  done  to 
determine  -the'  extent  to  which  the  needs 
of  employers  are  beijag  met  and  what  should 

*t>        bejdone,  i£  anything,  to  amend  the  pro- 

*  gram.  * 


3.  .  Objectives 

Your  objectives  should  be  deriyed  directly  from  the 
purpose  and  the  rationale.  \?his  section  states  fn  specific 
terms  what  it  is  your 'study  is  designed  to  find  out*  Your 
objectives  can  be  phrased  as  questions  orTas  action  sen-; 

tertges.        a  %  •  - 

phis  is  an  important  part  df  your  evaluation  plan  . 
since  it  is  here  tbat\  the  first  teal  structuring  of  the  / 
equation  "begins.     The  m^hodology  of  inquiry  and 'analysi^/ 
the  instruments  to  be  use.dr  and  the  persons  to  be  contacted 
.will  defend  on  ,your  objectives..    The  objectives  can  be 
changed,  added- to,  or  subtracted . from,  while  the  details 
-of  the  methodology* are  being  worked  out. 

Taking  4s  .an  example- the  OJT  P^  ject  mentioned 
above,  some  of  the  objectives  might  be: 

•  *  To* determine  the  extent  2nd  causes  or 

*     dropout,  absenteeism/  and  malinger in&S' 
'    in -the  project,  and  to  relate  the 

finding  to  the  nature,  of  recruitment 
'*  and  assessment 

r 

•  ,    To  determine  .what  the  participants  ex-  * 

pected^from  the  project  before  beginning. 
•    it  ii/  terms  of*  K  0  ^'  % 

-  the  difificulty  of  the  wotk 

■  -    their  ability  to  cope  with  the  work  n 

-  the  effect  of  work  on  their  health 
*                  -    what  it  would  do  for  the  com- 
munity ,  . 

%h  -   ~its  corftribution  to  their  r 
financial-  independence 
°  -    its  effect  on  their  general 

^  morale. 

•HI- 29  9  410. 


To  determine  participant  attitudes  *on 
the  same  questions  three  and  six  months 
later. 

To  determine  what  knowledgeable  staff 
of  the  employer  think  about  the  quality 
of  work  done  and  tjie  level  of  effort  ex- 
hibited by  project  paii^icij{ants. 


4.  Methodology 

>  After  you  have  set  your  -ewd^uaMon  objectives,  you  need 
*to  devise  a  method  for  realizing  them.     This  section- describes 
th2^  methodology.     It  should  include  a  description ^of; 

data  that  will  be  needed; 

me^ns  for  collecting  that  data; 

a  plan  for  pjjocessipg  and  analyzing 
the, data; 

•  '     a  scheme  for  presentence  the  data'  in 

a  meaningful  way,  pne  wfcrffch  will  pro- 
vide a  base  for  making  judgments. 

In  addition  to  the  above,  it  is  convenient  to  include 
^in^the  methodology  the  following  types  of  information: 

•  ~  Arrangements  which*  must  be  made  to 

carry  out  the  evaluatiqp,  . 

~*  *  *% 
0       Discussion  of  anticipated  difficulties 
arid  ways  of  meeting  th^m  \ 

•  Description  of  the  management  *  decisions 
necessary  to  implement  the  evaluation 

In  "describing  the  meana  of  collecting  datia,  the  following 
topics  should  be  addressed,  if  applicable: 

-  •       Statistical  techniques  to  be  used 

*  (sample  size,  statistical  tests,  con- 
fidence intervals,  etc.) 

•  . Questionnaires  to  be  administered  '  ✓ 

•  Interviews  to  be  conducted 
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#     •       Object ive^testing  to  be  done 

•  Records  to  be  reviewed,  r  * 

5.     Instruments  > 

This  section  identifies  all  instruments  which  you  will 
need  to  develop  to  implement  your  evaluation,  describes 
their  basic  characteristics  an^  the  process  which  will  be 
followed  to  develop  them.     Iteins  which  might  bfe  inducted 
here  are: 

•  questionnaires 

•  listfs  of  interview  questions  or  topics 

•  lists  of  observation  Criteria 

m       lists  of  special  data  to  be  collected 
from  project  operators. 


) 


6.     Work.  Plan  and  Schedule 

yoRr 


This  section  sets  out  the  timetable  for  your  evaluation/ 
Thig,  timetable  should  include:    gates  for  pre-testing  and 
post-testing;  dates  for  conducting  interviews  and  observa-- 
tions;  dates  ,  for.  ail  otlfer  activities  described  in  your 
design.    Additionally,  thS  timetaMe  should  include  dates 
for  submittal  of  evaluation  feports.     This  section  should  also 
specify  the  staff  to  be  assigned  to  the  evaluation,  their 
level  of  effort,  j^rtd  the  monies  and  other  resources  to  be 
committed  to  ensure  a  successful  evaluation. 

•  , "  *  >  ■ 

7.  Reports 

ft>  repeat,  this  sectitrT  indicates  the  types  of  reports 
that  will  result  from  the  evaluation.     It  should  specify  for 

« 

each  report  theV  * .      *    .  , 

•  use  of  the  report  -  -v. 

•  audience  for  whom  it  isinteiroed 

•  content^ 

4       •  format  ,    -  '  / 

*  '     •  submission  date.  j 
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CONCLUSION  "  ,  , 

In  cqnclusidn,  £our-  PIC's  'evaluation'  research  design  (plan) 
is  the  means  forgetting  that  information  which  you  will  need 
to  make  ma jor  progfcammatic^fecisions  and  to  change  program 
direction  and  operations  over  time,     A  similarly  constructed, 
plan  will  give  jjp*  the  data  necessary  to  make  those  decisions 
in  a  systematic  and  informed  manner. 


Mayor's  Office  of  Manpower,  City  of  Chicago, 
Planning  and  Evaluation  under  CETA  (Wash- 
ington, D.C.:  U.S.  Department  of  Labor, 
January  1976) . 
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Step  n 


Create  Res^jrch  and  Data  Gathering  Instruments 


»The  purpose  of  this  step  is  to  develop  all  research  and 
data  gathering  instruments  necessary  to  implement  your 
PIC's  evaluation  plar\.u  -  The .  npst  common  data  gathering 
instruments  are:     baseline  d^ta  worksheets/  9urvey  ques- 
tionnaires, and  interview  guides. 


t 


Baseline  Data  Worksheets  -  forms  designed  to  permit  uniform 
and  easy  analysis  of  data  gathered  routinely  through  PIC's 
management  reporting  system. 

Survey  Questionnaire  -  instrument  designed  to  collect  data, 
opinions,  or  attitudes  regarding  aspects  of  a  PIC's  program. 

Interview  Guide  -  format  which  provides  a  logically 
sequenced  and  comprehehsive  list^of  questions  regarding 
a  certain  area  of  PIC  performance  to  be  asked  in  a  personal 
interview. 


r 
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g(  U   linowledge/Cwbimtfes  Eating" 

Review  the  information  provided  above.  Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  .this  step -on  the  scales  below.  # 

Knowledge 
•  •  Capabilities 


1  

i 

1 

J 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

•Avg. 

1  ' 

 1  

» 

I 

Excel- 

Above 

Aver- 

Below 

Poor 

lent,- 

*    Avg . 

age 

Avg. 

Based  upon  your  ratings,  decide  whether  you  need  to  .read  the  cootent 
infprmation  on  this  step  which  follows  immediately  or  proceed  to  the 
next  step.  ■ 
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PME  PROCESS  NOTES/ ACTION  PLANNING  PAGE 


NOTES  (Points  that  are  important  which  you  want  to  remember) 


7 


^QUESTIONS   (Items^  on  which  yoif  are  unclear  or  want  to  get  additional 
^  \       infpr^iation)   >  .  '  v- 


innation, 

▼  # 


ACTIONS  (Specific  st^ps  that  you  intend  to  take  to  implement  your 
PIC's  PME  process  in  tjiis  area)  ^ 


Steps  > 


Resources/Persons 


Time  Frame 
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INTRODUCTION  •  • 

Your  PIC  .should  use  the  interim  between  planning- and 

actually ' implementing  its  evaluation  to  create  all  of  the- 

instruments  and  worksheets  to  be  used  for  the  research. 

Your  instrument  needs  will  be  dictated  by  the  objectives  for 

your  evaluation  and  the  data  collection  techniques  to  be 

employed.     If  your  PIC  restricts  its  evaluation7 to  the  outcome 

area,  the  primary  instrument^  you  will.haVe  to  create  will 

be:  *  m 

Baseline  Data  Worksheets  -  forms  J|esigned  to 

permit  uniform^  and  easy  recording  of  basic  4  < 

data  pfertaining  to  participant  or  project  * 

progress  for  analyzing  existing  data. 

If,  on  the  other  hand,  your  PIC  broadens  the  scope  of 
its  evaluation,  there  are  two  commonly-used  instruments  you 
might  chopse  to  develop  in  order  to  secyre  additional  data: 

Survey  Questionnaire  -  an  instrument  designed  V 
to  capture  data,  opinions,  or,  attitudes  re- 
garding aspects  of  your  PIC's  program.  The 
questionnaire  may  consist  of  forced-choice 
questions;  scaled  items;  or  some  other  form 
of  quantifiable  responses. 

Interview  GuiAe  -  a  formkt  which.,  provides  a 
sequenced  fend  comprehensive  list  pf  open- 
ended  questions  to  be  asked  in  it 'personal 
•^interview. 

The  characteristics  and  construction  of  these  instruments  are 
discussed  in  the' following  pagers. 


A.     DEVELOPING  BASELINE  DATA  WORKSHEETS 

— ■   1 

> 

As  noted,  if  your  PIC  decides  tq.  perform  only  an  outcome 
evaluation,  most  of  the  data  you  need  will  be  easily  retrievable 
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through  either  required  reports  generated  f;or  the  Department  of 
Labor  or  your  PlC's  internal  management  reporting  system. 
Therefore,  the  primary  data  gathering  instruments  you  will. have 

to  develop  are  baseline  data  worksheets.     These  worksheets  will 

•  *  • 

be  used  to  break  out  information  in.to  the  categories  and  against 
thdtfe  criteria  or  indicators  which  you  wa'nt  to  use  to  assess 
yrour  program. 

The  primary  criterion  that  you  should  observe  in  creating 
a  baseline  data  worksheet  is  that  it  must  have  the  capacity  for 
recorc^P^  and-  presenting  all  relevant  data  in  an  efficient  and 
usable  manner.     Items  normally  included  on  baseline  data  work- 
sheets for  participants  are:  ,  % 

•  demographic  information  (e.g.,  educatiqn, 
age,  ra^p/ethnicity) 

#.^/-  treatment  or  services  provided  % 

•  outcomes  achieved 


cost  information , 


EGS-type  itjms  normally  inc 


lucle: 


} 


demographic  or-  quantitative  information 
(e.g. ,  employers- contacted;  TJTC  vouchers 
issued;  interagency  -agreements  reached) 

products  or  results  achieved 

cost  information.* 


If  your  PIC  has  created  the  management  reporting  display 
formats  as  discussed  under  "Stage  1,  Step  3,  Design ' Management 
Reporting  and  .Monitor ing  System, "  then  ^you  can  use  the  same  or 
similar  forms  asjbaselip^*  data*  worksheets  to  record  and  anal^e 


data  for  summary  evaluation. ^  For  illustrative  purposes,  a 


/ 


simplified  version  Qf  a  basic  data  worksheet  for  analyzing  per- 
formance against  plan  or  standards  is  pre^nted  b^low. 


Project/ 

Standard 

Actual 

Variance 

,  Area 

Indicator 

or  Plan 

■  Performance 

.%  +  or  - 

4 

m 

f 

d 

) 
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B.     DEVELOPING  SURVEY  QUESTIONNAIRES  „ 
—  —  7   .  . 

The  following  discussion  highlights  basic  factors  whxWr 
your  PIC  shoUld  be  aware , of  in  creating  a  survey  instrument 
or  questionnaire.     These  factors' are: 

#  Format 

#  '  Language  ^j 

#  Layout 

^#  Pre- testing 

However i  developing  a  sound  survey  instrument  is  a 
science  reqijirina  knowledge  and  skills*  in  the. areas  of  survey 
research  and  questionnaire  design.    Jjjetefore,  if  your  PIC 
decides^  to  implement  a  survey  questionnaire*,  you  should  ensure 
that  you  either  have  or  can  secure  the  expertise  necessary  to 
make  that  survey  successful.     .   •  . 

Format  9  i 

A  questionnaire  is  basically  a  series  of  questions  or 
items  relctg,ng  to  subject  matter  about  ^which  the  potential 
respondent  may  ^reasonably  be  expected  to  Jiave#  opinijfcis,  atti- 
*tudesf  or  information. 

Survey  questions  may  either  be  forced-choice  or  open-ended 
Forced-choice  questions  list  a  full  range  of  possible  answers, 
^pne  of  which  should  best  represent  the  views  or  status  of  each 
respondent.     Open-end^d  items  ask  poi^gen^ral  questions  which 
may  require  expansive  responses.     Therefore,  possible  answers 

are  'not  listed. 

Af~§tre*tionnaire  may  contain  either  or  both  forced-choice 
and  o'^enrended  items.     In  addition,  the  instrument  may  include 
scaled  or- ranking  items.     These  latter  items  ask  the  respondent 
to  assign  a  numeric  value  to  his  or  hlr  response.     Examples  of 
these  three  types  of  questions  are  listed  on  the  following 
page.  • 
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m   :f  •        Example:    -Forced-Choice*  Item 

What  is  your  general  opinion  of  tfre  local  CETA  program? 
(Check  one, )  * 


,  Very  v         Somewhat  Somewhat  Very 

Favorable        /         Favorable  Unfavorable  Unfavorable 


^        ^      Example:.    Open-Ended  Item' 

What  is  your  general  opinionj  of  the  local  CETA  program? 
Discuss,  1  . 


%     Example:  ;  Scaled  Item 

What  is  your  general  opinion  of  the  local  CETA  program? 
(Circle  the  appropriate  rating.)  ■ 

i 

Positive  ^  Negative 


•V  %  S— — i— — H  '—i  


I 


The  advantages, of  forced-choice  and  scaled  items  is  that 
they:    -  4 

•  provide  discrete  response  categories  to 
permit  precision  in  capturing  respondent 
information;  * '  : 

* 

•  encourage  responses  because  they  can  be 
x  an  swerved  rapidly;  and  t 

|      •  m    terfd  to  compel  respondents  to  provide  • 
some  answer  or  rating  for  reach  item. 

Open-ended  questions  have  the  advantage  of  encouraging  more  , 
expansive  and  even  anecdotal  or  explanatory  replies.  .However, 
experience  sugges€fe  that  respondents  to  mail  questionnaires  are 

413 
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less  likely  to  answer  an  open-ended . than  a  forced-choice  item. 
•Open-ended  questions  are  also .difficult  to  quantify  and  compare 
with  other  responses.    The  types  of.  i^ms  to  be  used  in  a  • 
questionnaire  will  be  dictated  by  the  "nature  of  the  informa- 
tion requiredjThe  more  specific  discriminations  that  your 
PIC  needs  tc<be  Jble  to  make  in  areas  or  on  items,  the  more 
advisable  it  is  to  use  the  forced-choice  format. 

Language 

Regardless. of  the  type  of  questions  that  you  decide  upon, 
each  survey  question  should  be  as  di«  and  as  specific  as 
possible.     The  language  used  should  llRamiliar  to  the  ' 
respondents.     Jargon* and  multi-ay llabid  words  should  be 
avoided.  tTo  the  extent  possible^each  question  should  be 
worded  so  that  every  respondent  would  agree  on  its  meaning 
ana  interpret  it  in  the'  ,a»e  -way.  ■ 

Layout      „  *  *' 

Your  survey  instrument  should  be# conceived  as  a-complete 
entity.    The  .sum  of  its  items  should  have  the  potential  of 
Producing  the  full  range  of  information  desired  by  your  pic. 
Each  i««n  should  serve  a  purpose  and  the 'item  sequence  should 
provide  a  natural  progression  —  preferably  from  the  general  ' 
to  the  specific.     Typically,  the  first  one  or  two  items  on 
a  survey  instrument  should  be  general*  questions  designed  to  ' 
introduce,  the  subject  matter  and  gain  an  overall  impression 
of  the  respondent's  views.    Subsequent  items  should  deal 
with  significant  elements  of  the  subject  in  a  logical  sequence. 

Pretesting 

*      *->  * 
•   A  pre-test  serves  a  number  of  purposes  ^including: 

•  ensuring  uniform  understanding  of 
instructions,  items  /and  response 
categories 

•  •    providing  an  index  of  the  time  required 
for  the  survey's  completion 

'•     •       ,/  420, 
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confirming  Iwhether  or  not  the,  desired 
informatioii  can  be  acquired 

securing  aj perspective  on  the  advis- 
ability ofj  any  ^sensitive"  or  contro- 
versial itJems 


, •       obtaining ! an  informed  opinion  on  the  format 

and  appearance  of  the  instrument  as  well  ^  - 

as  of  thel sequence  of  *he  items. 

All  survey  instruments  should  be  pre-tested  prior  to  \ 
administration.     The  preferred  method  of  pre-testing  is  to  have 
'three  or  four  .members  of  the  target  group  complete  the  ques- 
tionnaire and  the/  discuss,  it  as  a  group  with  an  '©valuator. 
However,  in  the  event  of  a  smalf  target  group,  the  prp-test 
.can  be  administered  xto  nongroup  members  who  are  familiar  with 
the  program  being  evaluated.*. 


C .     DEVELOPING  INTF3VIEW  GUIDES  AND  PROCESS 


1 


| 

\ 
i 


An  interview*  guide -is  merely  a  specialized  form  of  a 
questionnaire.     Therefore,  the  same"  barsic  principles  which 
•apply  to  questionnaire  development  apply  to  the  interview, 
guide.    However,  there  are  some  particular  characteristics  of 
a  guide  for  a  personal  Intervi^.    These  characteristics  include 

•  »    an  introductory  statement  regarding  the 
,  purpose  of  the  evaluation,  and  particu- 
late the  interview 

•  opening  question^  that  are  general  in 
scope  and  whielT encourage  the  interviewee 
to  be  expansive  in  his  or  her  remarks 

•  a  flexible  ordering  of  questions  in  the 
event  thaythe  interviewee  addresses 
some  Gf  ifae  issues  before  they  are 
specifically  raised  by  the  interviewer  £  

•  a  core  of  key  issues  which  must  be  asked 
as  a  minimum  if  the  interviewee  is  being 

>  feomewhat  nonresppnsive 
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•  sufficient  blank  space  lLo  permit  the 
interviewer  to  record  responses 

#  concluding  remarks  t9  advise  the  inb£r- 
'viewee  of\  how  the  interview  data  will 

be  used  amTvto  answer  any  questions 
'   the  respondent  may  have. 

'    Given  the  fact  that  the  interview  is  a  personalized  inter- 
action ,  the  results  are  as  dependent  upon  the  method  employed 
by  the  interviewer  as  upon  the  Guide.     TherefQre,  jjjf  your  PIC 
decides  to  dp  personal  interviews  prior  to  the  condtct  of  any 
interviews ,  you  should  conv§ft£  everyone  assigned  to  do  the  - 
interviewing  to  discuss  the  process. 

In  this  meeting ,  the  interviewers  should  achieve  con- 
sensus oft  their  understanding  of  the  core  questions  that  must 
be  asked  and  the  style  in  which  the  interview  should  be  con- 
ducted (e.g. ,  formal  or  informal) The  interviewers  should 
also  agree  on  the  manner  in  which  follow-up  questions  sheuld 
be  asked,  in  order  to  expand  or  clarify  responses.  Finally, 
the  interviewers  should  review  a  prototype  interview  write-up 
so  that  they  understand  the  content,  levpl^Of  specificity, 
and  elaboration  expected  in  their  interview  reports. 


CONCLUSION 

There  are  other  research  instruments  which  your  PIC  might 
choose  to  employ  rather  than  or  in  addition  to  questionnaires, 
interview  guides,  and  baseline  data  worksheets.  However, 
the  general  criteria  for  developing  those  instruments  are 
similar  to  the  principles  stated  for  the  three  specific  types 
discussed  undier  this  step.     In  conclusion/  some  general  guide1- 
lines  which  your  PIC  should  follow  in  creating  evaluation 
research  instruments  include: 
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•  Restricting  the  data  collection  effort  to  <- 
that  information,  required  to  achieve 

the  objectives  of  the  evaluation         -  i 

* 

%  '     •       Ensurinq  that  your  instrument  is  compre- 
hensive enough  in  scope  to  answer  all 
*   questions  which  are  important  to  your 
evaluation  effort 

•  Making  the  wording 'of  items  simple  and 
nontechnical  so  that  they  can  be  easily  % 
interpreted  by  respondents/ 

s        1       Defining  technical  terms,  if  they  must  be 
^       used,  so  that  they  can  be  understood 
uniformly  by  both  the  respondent  and  " 
the  data  collector 

-    •        Desiqning  the  instruments  so  that  tjiey 
are  streamlined  in  format  and  easy  to 
use. 

Adhering  to#  the  foregoing  guidelines  will  greatly  enhance 
the  utility  of  any  special  evaluation  research  instruments 
that  your  PIC  has  to  create.  *  ,*  * 


Olympus  Research,  Self-Evaluation  of  CETA 
-.Manpower  Programs :     A  Guide  for  Prime  Sponsors 
(Springfield,  VA:  U.S.  Department  of  Commerce , 
1975). 

Fink,  Arlene,  and  Kosecoff,  Jacqueline, 
Evaluation  Primer  (Washington,  D.C.: 
Capitol  Publications,  Inc.,  1978). 
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G-     MflNfrjlCjfr  PROGRAM* 
t 


Purpose:  «To  assess  and  redirect 
as  necessary.  . 


program  activities 


1/  AnalAe-  Planned  versus  Actual  : 
1  ,Pj«<ormance 

2\yinitiate  Follow-up  Corrective 
Monitoring  *  i    •         •  .-. 


3.     Eftsure  Necessary  Corrective 
Actions     *■  " 
*  > 


> 
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Analyze  ^Planned  Versus  Actual  Performance 

The  -purpose  of  this  s-tep  is  to  assess  program  and  project 
activities  to  ensure  progress  toward  goals  and. to  identify 
areas  in  which  intervention  *  and/or  modifications  may  be 
required.     The  step  involves^  collecting , data  through  your 
PI£'s  management  regbrting  system,  and  then  arraying  ant 
'reviewing  ^^^^r^-^ 


"  4 


•  Monthly  Reports  -  basic  mpnitoring  data  increments  to  be 
submitted  by, all  PIC  project? • 

•  Control^  Limits  -  range  within  which  variance  from  planned 
,   level  of  performance  is  tolerable. 

•  Warning  I/Lqht-  -  indicator  that  project  has  exceeded  control' 
_ limit  <Qfc  a  given  area  of  *pref erence . 


TT 


■^l^dge/Capabi Ht1es  Rating 


Revie^  the  information  provided  above.  Rate  your*  PIG 1 s  knowledge]  and 
cc">*Mlities  with  reference  to  this  step- on  the  scales  below v 


•  knowledge 

1 

i 

-  1 

Excel^ 
*  lent 

Above 
Avg , 

Aver- 
age 

Below 
Avg. 

.  Poor 

•  Capabilities 

m 

1 

 1  

4 

lent  y 

Above 
Avg. 

Aver- 
age 

Be^ow 
Avg. 

Poor 

Bas6d  upon  yolar  ratings,  decide  whether  you  need  to  read  the  content  A 
information  on  this  'step  which  follows  immediately  or  proceed  to  the 


ext  step. 
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NOTES  (Points  that  are  important  which  you  want  to  remember) 


^gSTToNS^nHbemsonv^i^hyouare  uncledr  or  want  to  get  additional 
information)  ,» 


Steps 


Resources/Persons  .-  '* 
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INTRODUCTION 


T 


'  * 

As  discusrsed  earlier,  the  primary  ipfeans  to  monitor  your* 
PIC's  performance  is  by  measuring  the  relationship  between 
planned  and  actual  performance.  .  The  purpose  of  this  monitoring 
is  to,  assess  program  and  project  adtivities  on  an  ongoing  basis 
so  that*        \  '  ,  a    "  •      ~3  • 

•  problems         be  detected  asmn  as  ^ 
they  emerge 

:  #       sources  of  problems  may  be  isolated 

•  remedial  action  may  be  ils^Ltuted* 
while  the  program  or 4project^£s  stftll 
in^progiress  •  ^  ^ 

v/Your  PICfs  managemenf  .rfepoftin^  sy&tenJwill  provide  £he 
basic  information  reeded  tajteviep  performance  agaiffst  plan. 
By^ arraying  this  information, on  your  ^-ma^hagement  reporting 
dirflay  charts,;  ybu/wiil "b^  able .to  easily  identify  areas  in 
which  there  may  b^  problei^"/        \  \ 

A.    lookingIat  performance 

VERSOS  -PliAN  DATA 


The  specific  manner  in  which  yoi^^IC  lool^  at  performance 
verrfs  plan  {lata  will  be  detei#iBe£i  by  Kow^rou  have  divided* 
management  review  responsibilities*,  you*  ^management  reporting 

/stem,  andLmonitoring  plan.,    JJoweirer,  there  are  some  basic 
uechniq^es  or  steps 'that  yotir  PIC  might  want  to  employ  in  doing 
your  performance* versus  plan* analysis. 

v  * 

Performance  Versus  Pjlan  Monitoring  -  Overview 

^e£forniaflce'*ver^us[  plan  monitoring  is  primarily  based  .upon 

a  process  model/  for  analyzing  program  progress.     The  flow 

process  model  clA  be  applied  to  both  the  particiant  <*nd  the 

•  _  v  . 

EGS  side  of  your  program.. 

,*  • 
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Participant;    On  the  participant  side,  the  basic  flow  is 
defined  by/how  a  participant  moves  through  the  program.    This  is 
illustrated  in  the  figure  below.  v 


Participant'  Flow  Model 


Entry, 


I  On  Board 


Exit  1 


Flow  of  Participants 


Entry 


|  On  Board" 


Exit  | 


s 


As  the  figure  indicates, "there  are  three  basic  stages*  to  the  par- 

ticipation  flow  model: 

»  .     Entry         -  a  person  is  enrolled  in  a  specific 

project  or  activity 

#  On  Board    -  the  person  is  participating  in  that 

project  or  activity 

#  Exit  -  the  person  leaves  the  project  or 

activity 


EGS:     On  the  EGS  side,  the  basic  flow  is  defined  by  each  ... 

project-  itself.     As  on  the  participant  side,  there  are  three  major 

stages  to  project  activities: 

1  •  Design  -  project  plan  and  specifications  are 

developed  • 

•  Implementation  -  the  project  is  initiated  and 

activities/tasks  accompli3hed^ 

,         •  *      Completion  -  th6  project  is  concluded. 

  J  r  . 

-»  \  '  


Project  Flow  Model 


Design- 

\ 

Implementation 

 * 

Completion 

 * 
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Performance  versus  Plan  Monitoring  -  Application 

At  a  minimum,  your'PIC's  performance  ^rsus  plan  monitoring 
system  must  -include  the  necessary  measures       allow  you  to  monitor 
at  these  three  checkpoints  for  both  your  participant  and  EGS 
projects.    As  noted  earlier,  the  exact  measures  that  you  choose 
will  depend  upon  the  scope,  nature,  and  intent  of  your  local 
projects  as  defined  in  your  program  operation  plan.  The 
important  factor  at  this  point  is  ^ 

•  to,  get  data 

•  to  record  it 


^^taf 


•        to  Review  it  in  such  a  fashion  that 

it  enables  you  to  make  valid  decisions 
reqardina  Droaram  oerfor^ance. 


1 .    Obtaining  Data  -  Monthly  Reports 

Monthly  reports  are  the  basic  da£a  increments  for  your  . 
PIC's  monitoring  system.     These  reports  should.be  provided  by  all 

f  and  contractors  involved  in  either  participant  or  ECS  projects.  The  content 
of  these  reports  should  be  elements  directly  keyed  to  the  indicators 
or  measures  which  have  been  identified  as  central  in  the  r 
program  planning  process  ^nd  decided    on    in-  developing  yojfir 
PIC's  monitoring  plan.  '  * 

Participant-Related  Reports;     Items  or  elements  which 
should  be  included  on  a  report  from  a  participant-serving  ^ 
project,  using  the  participant  .flow  ^checkpoints  identified-  , 
earlier,  would  be:  t  %  \ 

#       On  Board:    Personjs  who  are  enrolled  in 
the  program  at  the  beginning  of  the 

month  *  . 

*  , 

m       Entries  (new  participants) :     Persons  who' 
'enroll  in' the  program  during  the  month 

m  Exits  (terminations) :  Persons  who  cease 
to  be  enrqlled  in  the  program  during  the 
month  for  any  reason,  including t 

Placements  -  persons  who  eftter 
employment  during  the  month  or 
who  enter  an  initial  employment 
period  —  . 
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~    Other  positive  terminations  -  persons 
who  transfer  to  other  training, 
return  to  school,  enter  the  military, 
or  go  on  to  other  unsubsidized  t 
employment 

-    Nonpositive  terminations  -  persons  • 
who  lcrft  the  program  for  reasons 
other  than  (a)  and  (b)  above 

Completers  in  holding  -  , persons  who  successfully 
complete  the  program  during  the 
month  and  are  waiting  for  an  appro- 
priate job  opening.         .  ' 


# 


Other  items  which  are  frequently  included  in  monthly 
reports  include  participant  characteristics  and  financial  measure 
ments  (accrued  expenditures  and  the  portion  of  expenditures 
transferred  directly  to  participant^) .  \ 

To  the  extent  possible,  you  sh6uld  track  common  dita  ele- 
ments for  all  those  projects  which  are  participant-^ep/lr\g  (e.g.v 
classroom  training,  OJT,  upgrading,  retraining).     In  addition) 
if  the  common  data  elements  do  not  account  for  important  services 
or  activities  undertaken  within  a  project,  then  supplemental 
elements  or  indicators  should  be  added.     Examples  of  such  indi- 
cators related  to  intakef'and  'assessment  are: 

*        •       number  of  counseling  services  held 
..  •      'number  of  assessments  and  tests  given, 
v  ' 

In  conclusion,  the  essential  consideration- for  reports 

from  your  participant-serving  projects  is  that  they  employ  common 

data  elements  supplemented  by  anj^sbt£er  measurements  particular 

to  each  project. 

EGS-Related  Reports.    The  majority  of  the  EGS  projects 
which  your  PIC  will  initiate  will  probably  differ  significantly 
in  nature  and  intent  from  one  another.    Thus,  it  is.  unlikely  that 
you  will  be  able  to  establish  common  data  elements  (e.g.,  entry, 
exit,- client  characteristics)  for  reporting  on  them.  ' 

However,  it  is  possible  to  develop^ a  common  monthly  report 
format  for  EGS  projects.     A  possible  format  for  reporting  on  EGS 

projects  is  presented  below; 

•  & 

%  *     s  III-320 
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Planned  Actual 


•  Products/Deliverables  Products  Produced  . 

Expected        .  '•    .  * 

•  Activities/Tasks  Projected  Activities/Tasks  Completed 

•  Results  Expected  Results  Achieved 

•  Projected  Expenditures  Actual  Expenditures 

The  format  above  relates  back  to  the  time-phased  flow  model  for 
EGSprojects  presented  earlier.    To  use' the  format  with  eaph  EGS 
project  that  you  operate,  you  determine,  using  the  pregram 

operations  plan,  the  indicators  which  are  appropriate  for  each  

area  and  then  have  the  operator  report  accordingly  against  those 
indicators.  . /  »  ~ 

2.    Recording  or  Charting  Data 

'  4 

As  yoi»r  PIC  gets  data  from  each  project,  you  must  con- 
vert it  to  summary  form  tp,  ensure  that  it  enables  you  to  review 
overall  program  performance  and  performance  by  components  or 
service' deliverers.  If  ypu  have  completed  the  earlier  step  on 
designing  your'PIC's  management  reporting  system,  then  you've 
already  developed  these  formats.  \ 

To  repeat,  thes#  formats  mu.st  be  laid  out  to  enable 
you  to  easily  review,  visually,,  total  current  program  activity 
and* to  compare  actual,  with  planned  performance.    These  fprmats 
can  employ  either  tables  of  numbers'  or  graphic  charts".    Your  PIC 
should  use  the  format  (s)  which  you  find  mos.t  utilitarian  for 
each  type  or  "revel  of'jprogram  review  that  you  mus-t  do. 


# 


3.     Reviewing  Data 

*  .  f 

The  primary  purpose  of  your  PIC's  data  review  is  to 
identify  areas  in  which  performance  differs,  significantly  from* the 
plan.    This  is  normally  done  by  assessing  variance  from'  the  plan 
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and  comparing  it  to  some  pre-set  control  limit  which  indicates 
an  acceptable  level  of  variance.    Since  control  limit  is 
a  central  concept  for  your  review yof  program  performance  data, 
let's  take  a  closer  look  at  it. 

Control  Limits .     Control  limits  are  designed  to  set  a 
range  of  tolerable  variation  from  planned  levels  or  scheduled 
activities  within  which  significant  monitoring  efforts  or  correc- 
tive action  efforts  will  not  be  taken.*    Control  limits  help  you 
to  concentrate  monitoring  time  and  follow-up  on  those  areas  which 
appear  to  be  critical.  ' 

There^ are  no  hard  and  fast  rules  for  setting  control 
,  limits^ — In  f act*.,  they  are  generally  set  #on  a  fairly  arbitrary 


basis.    Common  control  limits  on  the  participant  side  are  plus  and  minus 
10  or  15  percent  of  planned  levels.    However,  there  are  a  matter  of  different 
approaches  to  setting  these  control  limits.     They  include: 

•  Making  contrpl  limits  wider  during  initial 
phase-in  periods  and  then  narrowing  thgm 
as  implementation  proceeds.     For  example, 
you  might  choose  to-use  5 5  percent  plus 

or  minus  control  limits  for  the  first 
quarter  of  a  project,  reduce  the  control     m     t  »  / 
limit  to  plus  or  minus  15  percent  in  the 
.  second  quarter,  -  and  then  go  to  plus  or 
minus  10  percent  in  the  third  and  fourth 
quarters.  ^ 

•  •     .  Using  different  cbntrol  levels  for  programs 
of  different  sizes  or  for  subcontractors  to* 
Which  your  PIC  wishes  to  pay  particular 
attention.    For  example,  you  might  choose 
to  have  a  very  narrow  control  limit  on 
intake^  measures,  since  intake  which  is  either 
much/too  high  or  too  low  'will  produce  over- 
loads or  under enrtffciments  in  other  program 
components  for  several  mont.hs  afterward.  * 

•  Employing  narrower*  control  limits  on 
larger  programs.* *  As  you  can  readily 
realize,  applying  the  same  percentage 
variances  to  programs  of  different  Sizes 
can  have  radically  different  implications  ~ 
in  terms  of  your  PIC's  accomplishment  of 
its  total  plan'.     '  ' 
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•  Stipulating  narrower  control  limits  for 
problem  subcontractors. 

With  respect  to  EGS  activities,  control  limits  can  be  even 

more  arbitrary.     The -exact  limits  that  you  place  on  planned 

activities_or  deliverables  ^expected  or  projected  should  be 

♦  » 
determined  by  factors  such  as:  • 

»  *. 

,y   •       whether    the  activity  is  a  critical  pre- 
condition for  other  activities  to^occur 

•  .  the  amount  of  funds  involved  in  tjie  project 

t 

•  •  .  the  need  for  timeliness  in  project  com- 

pletion 

# 

,  -  -  #       the  public  visibility  of  project  results 
or  outcomes 

•  the  role  that  the  activity  plays  in 
directly  generating  or  creating  job 
opportunities  for  CETA-eligible  par- 
ticipants. •  ^ 

All  of  these  modifications  and  adaptations  of  the  control 
limit  idea  have  their  uses.'    For  monitoring'*  purposes,  your 
PIC  should  set  limits  to  reflecjb  its  particular  experiences, 
concerns,  interests  and  objectives.    As  noted,  areas  in  which 
these  limits  are  expeeded  will  be  those  4p  which  you  have  to 
do  follow-up  or  managerial  monitoring.  f 


B.     SOME  CAUTIONS  ON  PERFORMANCE 
VERSUS ~PLAN  ANALYSIS 

You  should  be  aware  that  performance  yersias  plan  analysis 
is  not  a  completely  perfect  system  for  monitoring  your  flC*s 
program  performance.    There  are  two  ma jot  limitations  on 
this  form  of  monitoring.     The  first  relates  to  the  form 
itself ,\  the  secon4,  to  the  prfmary  data  used  in. analyzing 
perfprmance.#  *■  / 

Plannihg, 

The^usefulness^of  performance  versus  p|.an  analysis  depends 

t 

in  large  part  on  the  soundness  of  the  originally  estimated 
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planned  actiyity  levels  or  project  schedule.     If  thes^| levels 
are  unreasonable — either  excessively  high/low  or  long/short-- 
the  value  of  subsequent  comparison  is  either  meaningless  or 
severely  compromised. 

Reporting 

A  performance  versi^  plan  system  can  be  no  better  than 

•  > 

the  data  on  which  it  relies.     If  projects  are, reporting  either 
inaccurate,  inconsistent,  or  invalid  da'taT,  then  ypur  review 
of  that  data  wiil  tell  you  very  little  about  actual  performance, 
and  will  not  serve  the  purpose  of  helping  you  to  identify  where 
program  problems  may  be  occurring.  11  ,t00^ 

Your  PIC  should  keep  the  foregoing  constraints  in  mind  in 
establishing  your  performance  analysis  procedures  and  in  re- 
viewing performance  against  plan*.     Irfthis  regard,  you  should 
make  provisions^  for  periodically  validating  data  submitted  by 
contractors -and  operating  units  and  be*  prepared  to  modify 
standards  upward  or  downward  to  make  them  more  realistic  and 
experience-based.  „ 


Olyi!!E>us  Research,  Self -Evaluation  of  CETA 
Manpower  Programs:     A  Guide  for  Prime 
Sponsors  (Springfield,  VA;     U.S.  Depart- 
ment  of  Commerce,  1975). 

Manpower  Administration  (now  Employment 
and  Training  Administration  [ETA]), 
•  Program  Asse^pment  Guide  (Washington f^5\C. 
U.S.  Department  of  Labor,  April,  1974). 
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Initiate  Corrective  Monitoring 


The  purpose  of  this  step,  is  to  determine  why  program  per- 
formance varies  from  planned  levels,  and  to  identify 
possible  remedial .actions  for  management  consideration.  A 
The  step  includes:    determining  that  a  potential  probl^A 
does  exist;  contacting  responsible  staff  or  project 
operators;  assessing  the  situation;  and  making  conclusion^ 
about  whether  corrective  action  is  necessary,  and,  if  so, 
what  types  qr  actions  might  be  appropriate. 


/ 


•  Corrective'  Monitoring  -  follow-up  analysis  to  determine 
why  performance  varied  significantly  from  planned  level 
-.and  tp  identify  possible  remedial  actions  to  correct 
situation. 


•  Management  by  Exception  -  approach  to. monitoring  which 
states  that  corrective  monitoring  should  be  restricted 
to  exceptional  deviations  in  performance. 


Review  tH^  information  provided  above.  -Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  this  step  on  the  scales  below. 


• ' Knowledge 


Capabilities 


I 

I 

1 

I 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

■ 

1 

i 

J 

Excel- 

Atjpve 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

Based  upon  your  ratings,  decide  wfiether  ybu  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or 'proceed  to  the 
next  step. 
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N^E^Tp^int^tha^ar^impo^  you  want  to  rememfrer) 


V 


QUESTIONS  (Items  on  which  you  are  unclear  or  want  to  get  additional 
~  information)  \^  ■ 


ACTIONS  (Specific  steps  that  you  intend  to  take  to  implement  your 
PIC's  PME  process  in  this  area)  ; 


Steps 


Resources/Persons 


Time  Frame 
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INTRODUCTION 

Your  analysis  of  performance  against  plan  will  enable 
you  to  pinpoint  areas  in  which  there  may  be  a  problem. 
However,  your  review  of  report  data  will  not  indicate  what 
the  source  of  that  problem  is.     TfSj&refore,  if  you  discover 
a  significant  deviation  fron^plan,  you  should  initiate  a 
more  in-depth  inquiry.    Appropriate  techniques ^include: 

*  on-site  monitoring  -  when  data  must  be 
collected  from  participant  or  contractor  W 
files,  or.  when  more  qualitative  informa-.         *  ^ 

^tion  is  desired 

•  interviews  -  with  participants,  employers, 
or  contractor  staff  to  collect  'data  which 
cannot  be  obtained  from  standard  sources. 

By  doing  this  follow-up  in  problem  areas,  you  can  determine 
whether  corrective  action  is  necessary  and  secure  the 
essential  information  for  preparing  a  corrective  action 
plan.    The  pages  which  follow  present  more  detailed  informa- 
tion on  steps  your  PIC  might  take  to^do  follow-up  :or  corrective 
monitoring  based  upon  performance 'versus  plan  analysis. 


,  A.     ESTABLISHING  A  CORRECTIVE  MONITORING  APPROACH 

c 

The  purpose  of  follow-up  or  corrective  monitoring  is 
,to  determine  why  program  performance  varies  from  the  planned 
levels  and  to  identify  possible  remedial  actions  for  management 
consideration.    There  is.no  single  definite  pattern  for 
responding  to  potential  problems  when  your  PIC  identifies 
them.    However,  t&t  following  sequence  Of  steps  constitutes 
a  basic  approach  which  your  PIC  might  employ,  modifying  it  . 
as  necessary,  to  do  follow-u^  or  corrective  monitoring-. 

#  « 

*  1.    Monthly  performance  versus  plan«da£a 
is  reviewed. 

2.    Projects  or  areas  in  whieh  control* 

limits  are  being  exceeded  are  E>in-  \ 
pointed. 
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3.    An  exception  report  is  filled  out  * 
on  each  area  in  which  the  variance  i.s 
judged  to  be  significant. 

■M.  These  reports  are* ranked  in  priority 
prder  and  then  assigned  to  staff  for 
follow-Up.  • 


5. 


8. 
'  + 

9. 


The  assigned  staff  person(s)  contacts 
the  projecy  operator  (s)  or  PIC  staff  involved  to 
set  up *meeting (s)  .  * 

The  situation  is  reviewed  by *the  PIC 
sta^  person  (s)  -and  the  project  operator (s) 
or  sfeff  to  discuss  possibly  .causes 
and  solutions. f  x 

Decision  is  reached  on  extent  of  + 
problem  and  on'ne^cL  for- further 

follOW^Up.  ~  *  ^ 

If  necessary,  additional  records^  ar$ 

reviewed  $nd/or  persons  are^contacted. 

«  •  •  - 

Based  upon  monitoring  visits,  a      .  " 
managerial  monitoring  report  is  pre- 
pared ^hich  states  whether 'formal         •  <$g 
corrective  actfion  is  necessary  or-  ♦ 
^whether  the  situation  can  be  resolved 
informally.     If  corrective  action  is. 
recommended^  the  itertltorincH  report  recomr 
mends  possible  steps  that  mightr*be  taken 
tb  correct  problem*  * 


HINTS  ON  DOING  ,CORRfeCTIVB%MONITORING 


Ag^.the  fforeg#ing  steps,  indicate,  the  mechanics  for  doing 
corrective  monitoring  are  not  compls^   However,  monitoring 
styles  vary  considerably^    Tp  do  the  mo|^  ef feet ive  ,  job  of 

Ltoringi  your  PIC  should  adhere  to_  the  following 


#   corrective  moj 


,  six  principle" 


rail 

i 


Manage  by  Exception         %  «  •  ^  v 

Lan jfeysus  performance  analysis  will  probably  identify 
ir  ,of  deviations  .frQm  your  PIC's  The  time- 

consuraing,  skill .demanding,  intensive  treatment  implied  by 
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corrective,  monitoring  cannot,  and 'should  hot/ be  given  to 
.  each  and  every  deviation.    Therefore,'  yqp  sfiould  control 
the  inclination  to  o  i  i  \Mm  jji  n       Yoij  5hould  restrict  cor-? 
rective  monitoring  to  exceptional  deviations  in  performance, 
It  should  be  focused,  on  Q^mLal  problems  and  bottlenecks. 


J*e  Systematic 

v      The  corrective  monitor  ooes  not  have  all  the*  time  in 
"•the  world  to  distinguish  cause  from  Effect.     The  objective 
of  corrective  monitoring  is  to  get  performance  back  to  or  up  to 
planned  levels  as  cfETT&kly  as  possibly.     In  order  to  ensure,  this 
end,  you  should-  have  a  definite  structure  and  focus-«-analytical 
framework — ;for  diagnosing  the  problem  area.    You  ^should  develop 
questions  which  wiil  allow^you  to  probe  into  those*  factors  or 
areas  whict;  you  think  might  account  for  the  problem.     Fot  example  < 
questions  to  ask  if  you  found  a  significant  variance  in  a  parti- 
cular project's  expenditures  might  include  thle^  following: 

Is  'the  variance  caused,  by  undefutili^%4iton 
'    (or  ovefutilization)  of  the  component?      .  / 

•        Is  the  variance  taiusdd  by  under§taf f ing 

(or  oVerstaf f-ing)Y  ,  * 


"  6  oft 


*m        Is  more  ior  less)  service  b&itig  given  y 
than  planned?  9 


.  m        Have  unit  costs  increased  for  decrea^d)? 

%  V 

•     »  Wre  the  planned  levels  underestimated  *  v 
.J(or  overestimated)?    J*lrat  walk  the  source 
of  the  ^efective  estimate? 

v      -  '    .  •  ' 

Use  the  Phone  First  ,  ^ 

Elaborate  systems  are  nqt  necessary  for  .taking  acti6n 
based. »upon  what  you  discover  in  your  plan  )jer.sus  p.erfqrmaflffce 
analysis.  '  I^you  find  some  items  of  significant  variance 
Within  an  area,  a  practical  approach  is  to  get  on  the  phone - 
an^  ask  the^project  director 'to' determine  if  the^|  is  some 
reasonable  explanation  for  the  variance.    You  can  use  such 
wu '  +  ar  call  t&:    .  # 

,  ■  430  -  * 

ERJC*      •  '     .  -  iu-32t 


•  determine  whether  on-site  itfonitorin,g. 

•  is  necessary 

•  identify  items  to  recheck  in  the  following 
month 

•   #       reduce  the  scope  of  ah  on-site  monitoring 
/  visit* 

Using  the  phone  can  save  time  and  strengthen  PlC-pro^ect 
operator  relationships.     The  calls  can  yield  explanations 
which  are  not' apparent  from. the' limited  figures  tracked  month 
to  month  and  can  also'  serve,  to  put^the  operator  on  notice 
with  a  minimum  of  embarrassment.  %f 

Pursue  a  "No  Surprises"  Policy  with  Operators 

The  words  "prbbleq£  and  "monitoring  visit"  can  traumatize 
project  operators.     There  a»  some  steps  though  that  your  PIC/ 
can'take  both  in^dvance  of,  during,  and  after  a  corrective 
fconftoring  visit  to  minimize  the  potential  negative  ramifica- 
ti5^?^^such  an%  intervention.'  ^ 

•  Make  sure  that  performance  standards 

are  clearly  stated  in  the  project  »~ 
operators  Contract. 

•  Advise  the  pro^ject  liaison  in  advance 
of .exactly  what  ar^as  you  will  want 

 *  ^J-lp^  at-  and  with  whom  you  will  want   _  J__, 

to  talk.  »  "    *  # 

J  •  •       Establish  a  definite  liaisoh  to 

coordinate  yotif  visit  and  a  firm  * 
schedule  and  set  of  procedures  for 
your  visit.  _ 

* 

•  .  If  you  need  to  conduct  personal  inter- 

views: *  » 

/  -         only  one  evaluator  in.  . 

an  interview;  no  "ganging  up"   *  ^  \ 

*  -  use  an  open-ended  questionnaire, 

passed  out  with  an  adequate  and  ^ 
honest  explanation  at  a  staff, 
meeting  • 

440  >  * 
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*    -if  possible,  conduct  necessary    f  *  * 

,      interviews  away  from  the  f aqilitx     s  , 
at  neutral  locations  and  at  times 
•  ,  and  places  most  convenient  to  tlje  •  w 

interviewee. 
;     #  "   '    Z  • 

•  *  Unless  it  i;s  essential  to  the  solution  of 

^       .a  problem,  avoid  making  judgments  regarding 
project  personnel.     It  is  normally  the 
fc      "         responsibility  of  the  project  operator 
i  to  make  staff  ad justments  to  correct  a 
' '  w'  problem.  - 

1  •       sLaore  adverse1  findings  through  your 

,  «       liaison  in  advance  of  disclosing  them 

•  ;      to  t)\e  Councils  '  * 

•  *    Give  project  operators  a  chance  *to  develop 

and  commit* to  potential  corrective  action 
steps  which  can  be \  incorporated  as  part 
of  your  monitQring  report.  - 

»■* 

-  The  Council*  probably  should  not  be  involved^wi^h  many 
monitoring  activities.    *  However ,%%"no  surprises"  applies  here 
as  well.    Where  problems  are  significant,  the  Council  should 
be  informed  so  that  i%  can  be  involved  in  determining  next 
steps.  ■  *  , 


Avoid  Unnecessary  Disruptions  to  Operations 

Your  PIC  ^-corrective  monitoring  activitynscah' inter- 

-f^re-^wi^thr-opera^ors  -carrying  out_.their_  missiojj .  Thi .  s  is_  

especially  true  for  those  projects  which' are  participant- 
serving.    Therefore,  you  should  ensure  that,  your  schedule 
•'fits"  best  with  the  normal  routine  and  operations  of  the  ^  • 
project*.  *  4 

Remember  -  The  Elan  Could  Be  Wrong  . 

•The  problem  could  be  with  the  plan  rather  than  the 
performance/    The  plan  is  not  scripture,.     It's  a  guide,  a  ^ 
goal,  and  can  be  changed  if  circumstances  warrant.  Therefore, 
.  you  should  always  reassess  the  validity  of  your  planning  * 
assumptions;  ahd  standards  as  an  integral  part  of  every  moni- 
toring effort.  '  -  ,  *\ 


As  a  final  point,  effective  corrective  monitoring  requires 
substantial  interpersonal  skill  and  programmatic  knowledge. 
Therefore,  the  managerial  ononitoring  assignment  should  not 
be  looked  upon  as  a  training  ground  for  new  PIC  staff.  With 
some  care  in  selecting  suitable  personnel,  adequate  planning, 
and  a  modest  amount  of  effort,  corrective  monitoring  can 
produce  significant  payoffs  in  terms ^of  more  responsive ^and 
,  effective  programs  for  your  PIC,  . 


Olympus  Reseferch, 
Manpower  Programs; 
(Springfield,  VA:  U 
1975*. 


Self -Evaluation  of  CETA 

A  Guide  for  Prime  Sponsors* 


S.  Department  of  Commerce, 


Manpower  Administration  (now  Employment  and 
Training  Administration  JETA] ) ,  Program 
Assessment  Guide,  (Washington,  D.C.:  U.S. 
Department  of  Labor,  April,  1974). 
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Ensure  Necessary  Corrective  Actions 


Thft^purpose  Qf  this  step  is  to  develop  and  implement  plans 
for- rcorrecting  project  operations  in:  those  areas  in  which 
significant  deficiencies  are  found.     The  step  consists 
of  two  parts:    develop  correct ive -action  plans,  and  monitor 
implementation  of  corrective  action  plans. 


•  Corrective  Action  Plan  -  report  which  proposes  method  for 
eliminating  performance  problems.  % 

•  Corrective  Monitoring  System  -  special  procedures  and 
methods  to  provide  feedback  oft  th§  status  of  the  imple- 
mentation of* a  corrective  action  plan.   \  ^ 


M(  ((  Know1edge/C>P*bnit^  Rating 


Review  the 
capabilit 


*  — » — 

e  information  provided  aboveS  Rate  your  PIC's  knowledge  and^ 
ies  with  reference  to  this  step  on  tjie  scales^  below. ,  r 


•  Knowledge 

•  Capabilities 


* 

1  li 

1  * 

V  1 

1 

Excel- 

Above . 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

1 

1  4- 

i 

J 

Excel- 

Above ' 

Aver- 

Below 

.  Poor 

lent 

Avg. 

>  age. 

Avg. 

-B^sed  upon  your  ratings,  decide  whether'you  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the 


next  step: 
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PME  PROCESS  NOJES/ ACTION  PLANNING  PAGE 


NOT 


SF^Poiht^^ha^^r^impo^ 


QUESTIONS   (Items  on  which  you  are  uncLear  or  "want  to  get  additional 
information)  * 


ACTIONS   (Specific  steps  that  you  intend  to  take  to  implement  your 
:    PIC's  PME  process  in  this  area) 


Steps 


Re  source  s/Persojis 


TjLme ,  Frame 


v. 
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INTRODUCTION 


Corrective  action  plans  are  developed  to  eliminate 
specific  deficiencies  in  your  PIC's  program  operations  which 
are  isolated  through  your  analysis  of  performance  versus 
plan,  and  then  confirmed  through  corrective  nfcfciitoring.  The 
plans  are  developed  ,in  response  to  and  based  upon  data 
gathered  in  the  corrective  monitoring  step.     The^  two  major 
t»sks  in  ensuring  necessary  corrective  actions  are  to: 

km       develop  corrective  action  plans 
•  '    monitor-  the  implementation  of  corrective 
action  plans.  t 


A      DEVELOP  CORRECTIVE  ACTION^LANS 

A  corrective  action  plan  should  address  each  area  of 
performance  where  significant  variance  occurred  and 
was  judged  to  be  a  problem  through  corrective  monitoring. 
These  plans  should  specify  the: 
problem  and  its  source 
extent  of  its  deviation  from  the  plan 
criteria  required  for  improvement  ' 
remedial  steps  to  be  taken 
time  frame*  for  improvement 
assignment  of ^responsibility  for  action. 

You  should  normally  develop -corrective  action  plans 
following  the  quarterly  review  and  analysis  of  your  PIC's 
program  performance.    While  performance  data  are  generally 
monitored  and  analyzed  on  a  monthly  basis,  it  is  normally 
more  useful  to  withhold  corrective  action  plans  until  more 
time  has  elapsed.    Preparing  corrective  actions  prans  on  a 
quarter ly *basis . 
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•  dilutes  the  influence  of  any  isolated  . 
v  monthly  aberrations  ,  H 

• .      provides  a  more  comprehensive  and 
,  representative  data  base  from  which 

to  initiate  action*  . 

•  alldw*s  you  to  take  a  more  thoughtful, 
analytical  approach  to  problem  remedia- 
tion. ^ 

Corrective  action  plans  should  be  negotiated  with  and 
approved  by  PIC  management  and  the  appropriate  contractor  or 
project  staff,  and  where  ^particularly  important,  the  Council.  The 
chief  criterion  that  you  should  employ  in  developing  those  plans 
is  that  they  remain  consistent  with  overall  PlC-program  goals, 
objectives  an*  ^tan^arc's.  • 

B.     MONITOR  CORRECTIVE  ACTIONS*- -r 
*  ~ ^  

*  * 

Corrective  action  plans  must  be  carried  out  effectively 
and  on  schedule  if  they  are  to  redirect  and  improve  program 
operations.     Therefore,  you  should  establish  &  special  reporting 
and  monitoring  system  to  provide  you  with  feedback  on  tfie  status 
of  the  corrective  action  plan  implementatidh.     Such  a  system 
involves  specifying  the        *  J 

•  typte  of  inforjnation  needed 

* 

\     •        schedule- for  implementation  " " 

t  r 

•  responsibilities  for  action.  * 

c        The  format  is  set  up  so  .that  follow-up  results  can  be 
noted  month  by  month  within  the  quarter,  until  all  th/ee  months' 
activitiej^  are  noted,  providing  soijte  indication  of  progress  9  . 
during  the  quarter.         t     *        ,  •  ' 

In  conclusion,  your  PIC's  ability  to  monitor  program  and 
project  progress  and  to  redirect  efforts  as  necessary,  will  < 
contribute  significantly  to  the  ultimate  success  of  your  PSIP. 
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Olympus  Research,  Self -Evaluation  of  CETA 
Manpower  Programst    A  Guide~for  Prime  Sponsors 
(Springfield,  VA:  U.S.  Department  of  Commerce, 
1975). 

•Manpower  Administration  (now  Employment  and 
Training  Administration  '[ETA] ,  Program 
Assessment  Guide  (Washington,  D.C. :  U.S. p. 
Department  of  Labor,  April,  1974).  t 
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H.     EVALUATE  RESULTS 


Purpose :        To  assess  the  effects  and  outcomes  of  the 
PIC's  program. 


1.  -  -  Collect  Data 

2.  Analyze  'Data 

3.,.  Feedback  Into  Planning  Process 


> 


r 
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Step  1 


Collect  Data 


The  purpose  of  this  step  is  to  collect  all  data  which  is 
necessary  for  evaluating  your'PIC'^  program.     Common  data 
collection  techniques  include:     baselinfe  data  retrieval;,  ' 
survey  research;  individual  interviews;  group  interviews; 
and  follow-up  studies.     Tw6  major  technical  issuer  asso- 
ciated with  data  collection  for  more  sophisticated  evaluations 
(those  of  the  impact  nature)  are  establishing  comparison/ 
control  groups,  and  sampling.  r 


Survey  Research  -  reliable  method/  employing  some  form  of 
questionnaires,  for  generating  data  pertaining / to  respondent 
experience,  attitudes,  and  opinions  related  to  an  area  or 
intact  of  PIC  performance. 

Follow-up  Studies  -  specialized  version  of  a  survey  or  inter- 
view, usually  used  to  determine  the  impact  of  a  tic's  program 
on  a  participant  over  a  given  time  period  after  complete/I  '  t 
(e.g.,  6,  9,  12,  18  months). 


•  Sam^Le  -  a  part  of  a  statistical  population  whos&^eharacteris- 
tics  or  opinions  are  studied  to  gain  information  about  the  whole 

•  Control  Group  -  group  which  does  not  participate  in  PIC  program 
with  characteristics  similar  to  persons  in  program  used  as 

^mechanisB^fo^contrasti^ 


/  . 


Review^the  information  provided  above.  >  £ate  your  PIC1 ^  knowledge  and 
capabilities  with  reference  to  this  step'on  the  scales  below. 


•  Knowledge 


•  Capabilities 


1  !  

 J  — 

i 

1 

Excel- 4 

Above 

Aver- 

Below » 

Poor 

lent 

^  Hvg. 

age 

Avg. 

1 

■ 

1 

1 

Excel- 

Above 

Aver- 

Below 

Poor 

lent 

Avg. 

age 

Avg. 

Based  upon  your  ratings,  decide  whether  you  need  to  read  the  content 
information  on  this  step  which  follows  immediately  or  proceed  to  the 
next  step.  ^  ^ 
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PME  PROCESS  NOTES/ ACTION  PLANNING  'PAGE 


NOTES  (Points  that  are  important  which  you  want  to  remember) 


QUESTIONS    (Items,  on  which  you  are  unclear, or  want  to  get  additional 
|  information) 


\ 


9 


ACTIONS  (Specific  steps  that  you  ihtend  to  take  to  implement  your 
PIC's  PME  process  in  this  area) 


Steps 


Resources/Persons 


Time  Frame 


ERLC 


III-342       450  ^ 


'  1 

'  INTRODUCTION 

The  last  step-in  the  preceding  stage  described  certain 
basic  tools  which  your  PIC  might:  develop  to  collect  and 
organize  evaluation  data.  "  This  step  suggests  procedures  for 
using  those  tools, in  the  process  ofidata  collection.     In  addir 
tiorr,  it  discusses  technical  issues  related  -po  conducting  a 
valid  and  reliable  impact  evaluation. 

<\  " 

% 

A.     ACQUIRING  EVALUATION  DATA 

9 

Ypur  PIC's  ^valuation  research  design  will  dictate  its 
dhta  gathering  jnethodologie^.     The  most  common  data 
collection  techniques  include:  .  %- 

•  Baseline  Data  Retrieval 

•  Survey  Research  + 

•  Individual  Interviews 

1   •  ; Group  Interviews  • 

•  Follow-up  Studies 

Baseline  Data  Retrieval  • 

Different  techniques  have  more  utility* for  the  different 
activities  that  they  are  to  evaluate.    For  example,  baseline 
data  retrieval  is  particularly  useful  for  training  activities. 
Survey  research  dan  be  useful  for  EGS  as  well  as  training 
activities.     Individual  interviews  can  be  ^particularly  helpful 
in  evaluating  EGS  activities.-  t  ^tk 

If  your  PIC  has  decided  to  do  "orily  an  outcome  evaluation, 
it  will  normally  require  no  more  data  collection  that  re- 
trieving what  are  available  from  existing  sources.  However, 

* 

baseline  data  arefr  also  the  foundation  for  a  relative  effec- 
tiveness* or  impact  evaluation.     Thus,  regardless  of  the  nature 
of  y9ur  evaluation. plans,  they  will  require  collecting  baseline 
data. 
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*  1 

1  •  standard  Sources  ^  • 

^  stated  in  Chapter  II,. your  PIC's  internal  management 
reporting  system  and  reports  prepared  fcor  the  Department  of 
Labor  'can  be  used1  to  secure  baseline  data  on ■  items,  such  as: 


participant  characteristics 
types  of  services  provided 
placement  data 
expenditure  information 
applicant  records 

•  '  participant  tracking  p 

prodiict  development  and  activity ' 
.accomplishment  * 

•  observations  and  monitoiy.ng  repo 
*  • 

These  baseline  data  should  be--collected  oh-an' 
for  projects  that  are  to  be*~evaluated .     In  this 

•  $ 

m        can  provide  leading  indicators  in 
.    .  advance  of  the  summary  evaluation 

undertaken  at  the  conclusion  o£  the , 
program 
4  .  * 

•  will  not  become  a  time-consuming  task 
for  the,  evaluator (s)  once  the  formal 
evaluation  gets  underway. 


2.    Back-Up  Sources 
. —   r-  

In  addition  to  the  data  wljich  can  be  retrieved  through 
standard  reports,  project  records  provide  an  excellent- source 
of  supplemental  information  that  may  be  of  value  for  .yo** 
4valu^tion.    This  is  especially  true  if  you  need  to  be  able 
to  m^ke  more^iscrete  judgments  among  projects  or  components  - 
in  terms  bf  tffeir  relative  utility*    Fc>r  example,  access  to  * 
original  data  sources  would  allow  you  to  track  back,  establish- 
ing a,  "paper  trail"  on  unsijccessfi^l  project  participants, 
including  items  such  as: 

•       demographic  ^characteristics 
*   *       length  of  time  in  project  . 
»»  \  treason  for  termination. 

This  typ^  of  information  can  be  extremely -useful  to  your  PIC 
iij  assessing  "how  well"  each  particular  project  performed. 


r 
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Surrey  Research 

r  Purvey  or- questionnaire  research  is  a  -reliable  method  of 

generating  dat,a  pertaining  to*  respondent  expediences, 
attitudes,  and  opinions  regarding,  your  PIC's  performance. 
Surveys  can  ascertain  respondent  views  or  information  bn 

4  any  sitaject  wi-th  which  they  can  reasonably  be  %xpected  to 
be  f  amiliar . 

Performing  survey  research  is  analagous  to  following 
arecipe:  *use  of  the  proper* ingredients  in  the  right 
amounts  will  yield  excellent-  results.     Trie ' essential  ingre- 
dients in  the  survey  research  rfeclpe  include: 

developing  the  questionnaire 
*  selecting  the  sample 
administering  the  survey 
analyzing  the  data 

terpreting  the  results. 


ERIC 


Questionnaire  development  has  been  discussed  under  the 
preceding  step.     Sampling  is  discussed  later  under  this 
step,  an£  data  analysis  and  interpretation  is  covered  in 
the  next  two  steps.     Therefore,  the  discussion  which 
follows* focuses  upon  the  actual  administration  of  the 
survey  .itself .  ^ 

0 ».  ^  survey  may  be  administered*  in  several  ways: 

•  fby  mail  »  a  ' 

m  .by -telephone  •  . 

f#%  *      m  irvperspn 

t  •  in  d  group.     -  •  ' 

*The  survey  may  alsc^ke  either  anonymouJ^  ot)  identifiable. 

Mail. surveys  require  detailed  a&d  exploit  {nstructions 
,  'because  if  a  respondent  has  any  "questions,  they  cannot  readily 
be  answered.     Because  these  surveys  may  be  completed  anony- 
' mously,  however,  they  have  the  advantage  of  anonymity  over 
personal  or  phone  surveys.    .Anonyntotts  surveys  encou^age^  * 
respondent  candor-,  clearly  an  asset  when  asking  probing  or. 
^personal  questions.    On  the  other  side  of  the  coin,  the 


I 
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response* rate  among  anonymous  respondents  to»a  mail. survey  1 
may  be  less  than  desired  and  thus  comprise  your  PIC 1 s  eval- 
uatidh  findings. 

The  ideal  mean's  l>f  .ensuring  a  high  rate  of  response  while 

•  'preserving  confidentiality  is  by  administering  the  survey  to 

respondents  in  a  group.    With  this  method,  questionnaires 
are  distributed  to  all  respondents  and  the  instructions  are 

*  read.    The  evaluator  answers  any  questions  and  then  the 
respondents  proceed  individually .%    The  unidentified  question- 
naires  are  collected  when  everyone  has  finished*. 

Ultimately,  the r method  .tyhfch  your  PIC  chooses  foj:  a  survey 
should  be  determined  by  three  fundamental  concerns: 

•       the  availability  of  resources 


the  feasibility  of  bringing  respondents 
together  in  a  group         ^  * 

the  need  for  a  good  rate  of  response 
(i.e.,  questionnaire  completion). 


Most  surveys  wil^L  not  produce  a  100%  response  rate.  For 
ejcample,  a  survey  may  have  a  target  sample  of  50  respondents. 
/Anticipating  that  some*  of  these  individuals  will  not,  for 
(whatever  reason,  return  a  completed  questionnaire  or  be  Willing 
'to  participate  in  either  personal,  .phone,  or  group  question- 
naire completion  sessions^^you  must  "oversample"  to  obtain 
the  50  needed  responses.     That  is,  questionnaires  should  be 
"administered  to  more  than  50  individuals.  m 
.  s      *     The  issue  here  is  one  of  anticipating  the  nonresponse 

ra£e.    ^This  should  be  done  based  bn  a  combination  of  experience 
^  -     and  iijtai^ion.     As  a  general  fule,  the  estimate  of  the  non- 

, response  *rate  should  be  on  the  generous  side.     An  even  better 
^tnile  is  to  know  the  respondent  group  well  enough  to  anticipate 
•  the* nonresponse  rate.    When.  50  responses  are  needed  for 

reliability,  it  is  better  to  receive  55  than  42  completed 
.questionnaires.     (See  discussion,  pp.   III-353-358  for  additional 
information  on  sampling.)  * 
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Individual  Interviews 

Individual  interviews  are  another  means  of  obtaining 

•evaluative  data  regarding  your  PIC's  performance.  These 
interviews  could  be  especially  useful  in  securing  informa-  - 
tion  on  the  EGS  sid'e  of  your  program.     Given  their  open-ended 

-  nature ,  personal  interviews  provide  the  opportunity  "for  the 
respondent  to  amplify  his/her  viewpoints  and  to-  ftfrnish  in- 
depth^  insights,  perspective^,  or  explanations  which  would 
not  be  possible  or  diff  i«ydt*  to  obtain  through  the  more 
structured  questionnaire  pOmpletion  fgjrraft. 

A  few. guidelines  yc*ur  PIC  should  follow  to  enhance  the" 

prospect  fo*'  a  successful  personal  interview  include: 

* 

'  General  . 

j 

m  Interviews'  should  t^e  scheduled  at  a  site 
where  the  respondent  will  feel  at  ease. 
•  If  possible,  interviews,  should  be  done  at 
*    ^the  respondent's  work  site. 

•  No  .interview  should  require  more  than 
one  hour,  and  the  respondent  should  ber, 
apprised  ofgthe  anticipated  interview  _ 
lengthfin  aavance.  ' 
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1  r  ,        "interviewer  Techniques 

The  interviewer  should:" 
4 

%      personally  contact  the  intetview 

subject  as  a  means  of  introduction, 
to  establish  rapport-,  arid  to  schedule 
a  mutually  convenient  time 

»  A  • 

»    ,  have  fail  recall  of  the  contents  of 

the  interview  guide,  and  be  as  familiar 
as  possible  with  the  subject  matter 

•  practice  using  the  guide  in  acfvanee 
4  of  the  J^jterview  session 

•  strive  to  put  'the  respondent  at  ease 

^at  the  otitset  of  the  session 
\  ^^^^ 

•  A  record. salient  points  on  the* inter- 
'  r       view  guide' |rtiile  simultaneously  § 

listening  to*  the  res^dndent  ^ 

*  111-347   .     ^OO  , 


restate  his  or  her  understanding  of 
particular  responses  at  frequent  'inter- 
vals for  confirmation  and  to  demonstrate 
th  the  respondent  that  his  or  her 
answers  jure  being  "heard" 
* 

ask  follow-up  questions  when  triggered  * 
by  a  respondent's  incomplete  ansyer 

encourage  the  relating  of  anecdotal 
information  as  a  means  of  explanation 
and  clarity 

ask  for»  and  &nswftr  any  questions  the 
respondent  maypjjjave  at  the  conclusion 
of  the  session  ^"^w 

prepare  a  formal  narrative  based  on 
his  or  her  notes  soon  after  the  inter-  ' 
view  has  beejv  completed. 


Group  Interva-^w^ 

A  group  interview  is  an  inexpensive  and *eff ectiveXmethod 

which  your, PIC  bould  employ7  as  an  alternative  to  a  aarftfes  of 
«  i 

individual  interviews.     These  interviews  typically  genei 
a  substantial  amount  of  useful  data.     The  basic  requirements 
for,  a  group  interview  are  tha%t  the: 

•  prospective  respondents  must  be  ,    '    %  - 
relatively  equivalent  in  terms  of* 

0       their  knowledge  and  experiences  in 
the  area  or  subject  matter  to  be 
discussed  and  evaluated 

\ 

•  respondents  jmist  be  willi«9-^o 
participate  openly  in  a  .group  inter- 
view. '  •  — 

With  regard  to  the  first  requirement,  the  respondents 
must  be  individuals  whQ,  for  example,  participated  in  the 
same  training  program  or  who  were  placed  in  similar  jofrs. 
In  terms  of  the  evaluation,  they  must  therefore  share  the 
same  principal  characteristics.     The  second  requirement  refers 
to  the  fact  that  the  participants  must  understand  the  nature 
of  the  group  interview  setting  and  be  ready  to  express  their 
opinions  candidly  in  the  presence  of  others. 
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A  group  interview  proceeds  in  the  following  manner: 

•  the  assembled  participants' are  intro- 
duced to  the  purpose  and  process  of 
the  group . interview  * 

> 

•  an  interview  guide  is  distributed  to 
all  participants 

•  the  guides  are  completed  independently  * 
-  by  the  respondents  . 

•  v  each  individual  then  reports  his 

or  her  perceptions,  as  noted  on  the 
interview  guide,  to  the  entire  group 

•  all  responses  are  recorded  on  flip-  * 
chart;  pap«r  by  the  facilitator- 

t 

^        •       there  is  then  a  group  discussion  to 
clarify,  combine,  and  supplement 
•    the  data 

»  . 
•     »if  concurrence  is  desired,,  the  par- 
r        "bieipants  are  asked  to  vote  to  deter- 
mine- areas  of  agreement  or  to  establish 
rankings  of  particular  itenis. 

The  product  of.  this  process  is  a  large  body  of  data, 
for  each  item  on  the  interview  guide.     The  opportunity  for* 
respondents  to  review  and  discuss  their  own  as  *ell  as  others' 
answers  serves  to  generate  more  comprehensive  and  thoughtful 
responses  than  are  generally  accessible  through • individual 
interviews.  . 


V 
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Follow-up  Studies  '  • 

Follow-up  studies  are  a  specialized  version  of  a  survey 
or  interview.    They  are  a  commonly  used  method  for  assessing 
impact  on  participants.     I„  addition,  they  can  also  be  employed 
to  ascertain  employer  satisfaction  with  your  Pic  programs. 
These  studies  can  be  highly  useful  means  of  evaluating  a 
project's -or  you?  Pic 's  impac^  after  a  participant  has  conf-  ' 
pleted  the  program,'  or  a  project  has.  ended.     Such  studies  can 
determine  whethfer  or  not  a  program  has  a  continuing  efSeot  on  , 

the  participant.  • 

<.        •  « 

can  be  used  to  determine  a  number  of  / 


Follow-up  studies 
things  including: 
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Earnings  improvement 

improved  prc^ortion  of  time  in  employment 
Increased  lafc>r  force  participation 

Movement  to  growth  occupations  and  industries  ^  * 

Movement  to  more  highly  skilled  jobs 
Movement  t6  more  secure  occupations  and 
industries 

Improved  jobw finding  and  job  holding  *  , 

competencies  9 
Improved  conformance  to  work  rules  and  mores 
Decreased  antisocial  behavior  , 

Improved  self- image  ^  ^ 

Differences  in  Outcomes  associated  with 
varying  participant  characteristics  (e.g., 
employment  status  by  sex  and  race,  or  wage 
changes  by  sex) 
•         Participants'^  employers'  views  of  the 
quality  and  impact  of  the  program! 

There  are  four  questions  which  ar*  critical  to  the  design 

of  a  follow-up  'effort; 

•  What  infdfmatio^  should  be  collected? 

'   n       •      How  man/affected  individuals  (clients 
or  employers)  and  comparison  group  per- 
s6ns  should  be  followed?  L 

•  How  long  should  they  be  followed?  At 
what  time  intervals? 

•  HOw  will  they  be  contacted? 

■  • 
1.    Deciding  What  Information  to  Collect 
•     The  first  step  in  a  follow-,*  study  is  to  decide  what  questions 
you  want  to  answer.     These  questions  determine  the  focus  of 
your  study  a*d  identify  the  information  you  will  need  to  collect. 

The  following  data  are  those  required  for  comprehensive 
follow-up  evaluation  of  clients.    Some  of  this  data  is  compiled  at  the 
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time  of  program  entry  and  termination, 
through  post-program  questionnaires. 


Other  data  are  secured 


Evaluation  management  xiata 

•  Respondent  identifier s;  i . e . ,  name ,  Soc ial 
Security  number,  and  so  forth  * 

•  Respondent  locator;  i.e.,  address,  tele- 
phone number,  a  contact  person,,,  and  so 
forth  ,  . 

•  Type  or  method  of  contact;  i.e.,  mail, 
telephone,  personal,  and  so  forth  p 

Demographic  attributes  and  variables 

•  Standard;  i.e.,  age,  sexjw  and  ethnicity 

•  Household  responsibilities;  i.e.,  marital 
status,  head  of  household,,  dependents, 
and  so  on  -  a 

m  Special  group  identifiers;  i.e.,  public/ 
assistance  status,  veteran  status,  handi- 
capped* and  so  on 

Program  and  service  descriptors 

•  Major  program;  .i.e.,  on-*the-jdb  training, 
classroom,  basic  education,  public  service 
education,  and  so  on  ^ 

•  Services,  received;  i.e.,  component  , 
•descriptors  » 

•  Length  of  service;  i.e.,  time,  service 
units  and  so  on.  * 

•  Service  provider  identifiers 

Work  history  description 

For  prg-program  and  post-program  periods, 
chronological  lists  and  descriptions  of 

periods  xrt^.  ~  

Employment: . .employed  period  descriptions 

%£job  finding  activities. .  .'friends ,  .  - 
'want  ads,  door  pounding,  public 
agency,  private  agency,  and  so  forth 

-  Employer  identifiers. . .name,  address, 
telephone  rJumber,  standard  industrial 
classification  (SIC)   code,  and  so.  forth, 

-  Job  descriptors. . .Dictionary  of  Occu- 
pational Titles   (DOT)   or  other, 
unionized,  skill  level,  and  so  forth 
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Wage  and  benefit  quantifiers. . .dollar 
values 

-  Work  intensity. . .full  or  part-time 
;  hofcrs  worked ,  and  so  fo Ah 

-  Perception  of  supervisor 

-  indicators  of  internal  mobility .. .wage 
change,  job  change,  promotion ,  and 

so  on 

-  Duration  of  employment 

-  Reason  for  termination  if  applicable 

Unemployment. . .period  descriptiohs 

-  Job  finding  activities ...  friends, 
•want  ads,  door  pounding,  public 

agency ,  private  agency,  and  so  forth 

-  Sources  of  income  dujsincf  unemployed 
period 

-  Duration  <?f  spell  of  unemployment* 

■* 

Not  in  labor  force. . .period  descriptions 

-  ^ptivity  while  not  in  the  labor  force., 
in  school  *  military,  pregnant,  house- 
hold responsibilities,  discouraged, 
prison,  and  so  ^orth 

-  intention  to  participate  in  labor 
force 


Income  descriptors 

•>  ScJUrces  of  income;  i.e.,  wages  and  salaries, 
public  assistance,  unemployment  compensation, 
and  so  on 

#  > 

•  Amount  of  income  ~  < 

*  «  •  »• 

m 

Attitudinal  and  behavioral  indicators 

•  Perceptions  of  job  obtained  • 

•  Perception  of  training  received 

•  Recommendations}  i.e.,  services  quality", 
content,  and  so  on 

•  Indicators  of  antisocial  behavior;  i.e., 
arrest  record,  drug  use,  and  so  on 

•  Knowledge  of  manjpo^tf?  services  available 

•  to  participant 
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Indicators  of  employers'  perception  of  clients 

•  Conformance  tb  work 

•  Productivity 

m  Social  acceptance  in  the  work  environment 

6 ' 

As  tlib 'Tfbregoing  list  of  potential  information  suggests, 
Bhe  most  critical  part  of  doing  a  follow-up  study  is  %q 
determine  what  information  is  important'.     As  reiterated 
throughout  this  Guide,  your  PIC  should  decide  ithis  based 
upon  your  particular  information  and  decision-making  needs. 
A^tin,  the  caution  is  to  restrict  your  data  collection  to 
only  that  information  which  is  critical  for  management 
purposes. 

2.     Determining  Whom  To  Contact  . 

One  of  the  most  important  questions  in  follow-up  designa 
is  determining  the  proportion  of  individuals  (i.e.,  partici- 
pants or  employers)  to  follow  (and  the  size  of  the  comparison 
grQup  if  tHer\e  is  one)  .     The  critical  variable  is  the  degree 
of  confidence  that  your  PIC  wishes  to  have  in  the  findings  . 
and  how  group-specific  the  findings  need  to  be^ 

•      One  obvious  alternative  is.  to  follow  up  on  all  affected 
individuals  and  on  a  comj^a'rison  group  of  the  same  size.  This 
solution  -minimizes  th£> confidence  problem.     However,  the 
obvijjj&s  drawback  is  that  you  seldpm  have  t*e  resources  for 
lOO^ercent  follow-up,  and  even  if  the  resources  were  avail- 
able, the  difference  between"  bfeihg  80  percent  confident  and 
95  percent  confident  might  not  be  justified* by  the  cost. 
Therefore,  except  for  very  small  programs,   sampling  will  be 
the  only  possible  approach,  and  it*  may  be  the  preferable  one  , 
if-  th^  sample  is  of 'the  apprbpriate  size/  is  randomly  chosen, 
aj^is  ablq^to*  be  contacted.     General  principles  for  sampling 
a^  discussed  later  under  this  step.  .  , 
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3.  .  Selecting  Follow-up. Intervals 

After  your  PIC  has  decided  whom  to  follow,  tha  next 
decision  is  for  "how*  long.  H    The  most  frequent  follow-ilp 
intepiais  in  shorter- term  manpower  evaluations  have  been 
thirty,  skxty,  ninety,  180  and ,365  days.     One  possibility  is 
that  you£/PIC  might  want  to  make  your  interva/s  correspond  to 
those  .used  in  national,  state  or  local  longitudinal  studies 
in  order  to  permit  easy  comparison  between  the  results  of  * 
your  programs  and  those  of  others. 

You  definitely  should  ensure  that  your  fijial  follow-up 
point  is  not  too\lose  to  program  completion.     As  a  rule  of 
thumb,'  a  final  follow-up  point  between  9  to  15  monttis  after 
program  completion  seems  Lq^  be  an  appropriate  distance 
to  assess  program*  effect  with  some  degree  of  accuracy  and 
confidenqe. 


v     4.     Selecting  Follow-up  Methods 

There  are  also  significant  choices  your  PIC  has  to  make- 
among  personal  and  telephone  interviews  and  mail  surveys  as 
means  for  follow-up.     Each  of  these  approaches  has  positive 
and  negative  attributes.     Phone* interviews  are  low  cost, but 
tend  to  produce  higher  response  rates  from  successful  former 
enrolLees.     Mailings  have  a  similar  bias.     Response  rates 
generally  fall* between  3D  and  60  percent,  depending  on  the 
use  of  incentives  for  returns  and  persistence  of  successive 
waves* of  mailings.     Face-to-f$ce  fpllow-up  generally  yields 
higher  response  rates  but  i#  more  costly  than  letters  pr 
phone  calls.     Most  sophisticated  f Allow-up  efforts  combine 
the  three  approaches,  saving  face- to^f ace  follow-up  for  indi**- 
viduals  -iWi#  do  not  respond  to  mailings. 

In  conclusion,  as  this  lengthy  discussion  testifies, 
follow-up  studies  require  a  great  deal  of  time,  effort,  and, 
resources.  ,  However,  the  return  to  your  PIC  on  this  investment 
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is  tha^t  the  findings  from  follotf-up  studies  will  allow  you 
to  tell  with  much  greater  precision  exactly  what  differences 
your  projects  and  programs  are  making  within  the  local  com- 
munity. 


B.      "IMPACT"  DATA  COLLECTION  - 
ADDRESSING  TECHNICAL  ISSUES 


,  „      'Regardless  of  the  specific  data  collection  method  chosen 
, or  the  aj:ea(s)  to  be  studied,  once  your  PIC  decides  to  do 
an  evaluation*  of  an  impact  nature,  there  are  a  number  of 
technical  concerns* associated  with  doing\^he  evaluation 
"right."    Two  of  .these,  concerns  are:  V 

•  sample  construction 

•  comparison  and  control  groups. 

The.  following  pages  provide  k  brief  introduction  to  these 
.topics.     A^'  has  been  noted  earlier,   if  your  PIC  ihtends  to 
go  beyond  outcome  or  relative  effectiveness  evaluation,  you 
should  refer  to  the  references  listed  under  Step  2,  p. III-370 
and  secure  additional  assistance  as  you  feel  necessary. 

Sample  Construction 

Several  types  of  research  necessitate  the  construction 
of  a  sample  on  which  the  evaluation  will  concentrate.  Sample 
are  used  primarily  in  survey  research,  but  they  <are  also 
ij^ded  for  such  data  collection  methods  as  a  review  of  case 
files  or  determining  whon  to  interview. 


1.     Types  of  Samples 


The  four  most  common  types  of  samples  are  the; 

•  .random  sample 

•  systematic  sample 

•  stratified  sample 

•  cluster  sample 


Random  Sample.     A  random  sample  is  one  in  which  every 
member  of  the  population  has  an  equal- chance  of  being  included 
in  the  sample.     Simple  random  samples  re,ly  on  the  use  of  random 
numbers.     These  may  be  obtained  from  random  number  tables 
available  in  a  number  of  statistic' books  rat  your  local'  library. 

The  advahtage  of  a  random  sample  is  that  no  other  factors 
need  be  considered.     For  purposes  o£  the  research,  all  members 
qf  the  population  are  equivalent.     However,  there  are  many 
instances  when  a  randQm  sampling  method  is  impractical  and 
other  sampling  Methods  must  be  used.     These  other  methods 
are  described  below.' 

Systematic  Sample.     With  systematic  sampling,  -members  are 
purposely  selected  from  all  parts  of^ a  population  in  a  con- 
trolled, structured  manner  which  assures  that  each  member  has 
an  equal  chance  of  selection.     For  example,  consider  a  sample 
of  individual  case  files-.     From  a  population  of  5,  000  folders, 
we  choose  a  random  sample  of  27  0.     The  first  step  is  to  deter * 

mine  the  "skip  interval"  by  dividing  the  population  number 

m 

by  the  sainple  number: 

5,000    t     270    =  18+ 

•  4 

'   The  second  step  is  to  find  a. random  starting  point  from 
among  the  first  1&  folders   (using  the  random  number  table) . 
From  that  first  number,  we  would  then  take  every  18th  successive 
folder.     This  process  would  produce  a  sampl^%f  27  7  (instead 
of  270) .     However,  as  noted  earlier,  oversampling  can  only 
further  enhance  the  reliability  and  representativeness  of  the 
sample. 

Stratified  Sample.     A  stratified  'sample  divides  the 
population  into  distinct  subgroups  which  are  then  sampled 
separately-     Stratification  would  be  appropriate,   for  example, 
when  you  want  to  look  at  various  components  or  projects  which 
are  unequal  in  size  or  characteristics.     Clearly,   it  would  be 

•# 

? 
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inaf^opriate  to  give  an  equal  chance  of  being  included  in  a 

sample  to  participants  from  a  large  classroom  training  com- 

ponent:#of  500  as  opposed  to  those  from  an  OJT  component  of  50. 

Therefore,  these  projects/components  could  be  stratified  'based 

on  component  size  and  any  other  defining  characteristics 

deemed  essential  and  then  the  sampling;  would  be  done.  The 

use  of  this  technique  enhances  the.  reliability  and  validity 

of  the  survey  findings. 

■ 

Cluster  Sample.     In  cluster  sampling  the  population 
is  conceived  as  being  made  up  of  a  number  of  clusters, -^each 
of  which  contains  sampling  units.     For  example,  consider 
that  the  OJT  participants  employed  by  jnedium-size  firms  are 
to  be  sampled  and  interviewed.     The  OJT  participants  are 
clustered  by  the  firms  that  employ  them.     A  cluster  sample 
would  be  obtained  by  random  sampling  an  adequate  number  of 
firms  and  then  interviewing; all  employees  at  each  one  selectee^ 
Cluster  sampling  in  this  case  would  be  of  practical  advantage  % 
in  saving  time  and  effort  in  traveling  and  in  settyig  up 
arrangements  for  interviews. 

Cluster  sampling  is  most  effective  whey  (1)  the  differ- 
ences within  clusters  are  as  large  as  possible   (i.e.,  each 
cluster  can  be  thought  of'  as  representative  of  the  entire 
population);  and  (2)   the  differences  between'  clusters  are 
as  small  as  feasible. 

2.     Structure  and  Size  of  Sample 

The  structure  and  size  of  a  sample  is  basically  dependent 
upon  two  variables:  *  • 

•  the  sizp  of  the  universe  j(i.e.,  thet 
total  group  from  which  the  sample  is 
to  be  drawn  or  selected) 

•  * 

•  the  characteristics  of  the  universe 
(i.e.,  those  traits  or  factors  which 
distinguish  the  group  from  any  other 
group) .  .  , 
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Characteristics  of  the  Univer^fe.     The  traits  that  dis- 
tlBaguish  cjfie  particular  universe  from  any  other -have  irtuch  to 
do  with  Xti  total" size.     The  more  discrete  the  traits,  tie 
smaller  the  universe.     For  example,  if* the  universe  is  com- 
*c      .     prised  of  "surviving  -•ex-Presidents'  of,  the  United  States*" 

its  popula         Would-  nuigljer 'three.     However,  if  'the  only  .trait 
^  is->that  of  "EarthlLing,  "  the  universe  ^buld  nvujjber  four  billioij. 

In  terms  of  ydur  PIC's  program,  the  universe  would,  *at 
-  $     a  minimum, ^be  restricted  to  individuals  whq»  participated , in  ' 
v    the  particular  project  (-s)  to  be  ^evaluated      If  that  prograjn 
.   "  haft,  different  components   (esg.,  Qj£,  classroom  training, 
c    world*  of  work  orientation,  placdlff&ty,  yoju  might^w^jit  to  focus 

on  each  of  these 1  subgroups.  *  in  addition,  you  would  want' t6 
f   know  the  basic  demographic  characteristics  (e.g.     sex,  educa-' 
— tional  background/ worx  history)  of  the  participants  in  each/ 
S  project.  ,      V  (/  »  * 

»  ^P^e  you  Jiave  dJetermiried  the  ±>ajsj.p  size  and  character^-,  * 

tics  df  your  .potentjia^l  Sample  utjiverse^  the  next  step  is  to 
make  your  sample  "reliable"  and  "representative."   '  *    •  ' 

*  representativeness  of  the*  Scfcple.-   The,  primary  criterion    ,  .4 
that  a  sample  must  meet  is  one 'of  representativeness:     This  * 
*  means,  that,  as. much  as  possible,  y©^r 'samal6  must  be  drawn 
to  match  the  Characteristics  of-  tKe  universe.  •  "  *. 

**      ^For-  example,  if  a  group  of  60  people  are  enrolled  in 
£;  classroom  .training,  *you  should  identify  thejftge,.  gender , 
^    educational  background,  and  employment  history  pfeach  par-'  4 
ticipant^.^JI^^*--©^^!^  trainees^  ari  men,  €0%  pf  th#  sampled 
,   •    should  .likewise  consist  of^jjen'.     If  70%  of  the  trainees  had 
^  previous  work  experienceN$/xh^%§Snftp^e  should  reflect  th&t^>ro-  . 

portion.  *  •/  / 

'    *  f  ~  , 

Reliability  ^f  the  Sample.  *  Reliability  is"tjie ^degree  of 

confidence  that  ybu  can  have  that  your  *  findings*  from  your  sample  - 

repj^senV  acc^iratfely  *what  you^ould  find  if  you  surveyed  your 

t#tal  universe.     The,  size  of  a  sample  that^you  will"  have  to  draw 

.  v  .  (    ■  *,  ■    '  „.  . 
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will  depend^ipon  how  confident  you  want  to  be  in  your;  findings- 
In  general,  the  Aore  representative *that  you  can  make  your 
sample^  the, smaller  it  needxbe^ 

The  following  guideline^ wil I  be  useful«An  determining 
a  sample  size.     These  guidelines  presulne  that  ^taryl&rds -of 
representativeness  are  met.  m  *  "  1 


 ;  

— • 

SUGGESTED  SAMPLE 

SIZES 

No .  in 
Universe 

'No.  in*  " 
^  Sample 

1  % 

of 

Universe 

1000 
500 
.  300j^ 
100 
50#  ' 

100 

i.                   60  > 
9        ,  • .    50  * 
20  « 
<*   •  15 

10%.-  • 

12% 

17% 

20% 

30% 

f 


r  4 

^  It  is  important. to  note  that  tte  'foregqing  guidelines./* 

:  are  general  and  not  absolute.     Prior  to  structuring  a  sample 
^*for  your  .  PIC '  s  .project,  you  should  xe^r  *to  *any  good  "statis- 
tics.te^.-  ^ke  texts  will  j^rovide  the  essential  instructions 
<  for  sound*  sample  selection.  #  ^ 


3.     Essential , Steps  in  Sample  Development  \ 

Inj  summary,  •  regarchess  ojE  'the  type  of  sample  that  ybur 
PIC  'dec/ides  to  undertake,  the  process  of  sample  development 
can  be  reduced  to  the  following* essential  step§.  w  * 

.•  4  r  Determine  the4  general,  population  of  .  4 

interest*  *  *) 

.  •      Determine  the  subgroups  within  that  •       *     •  * • 
, %      general  population  of8 interest  on  which  ; 

separately  identifiable*  findings  are 
.'z~~~Tcle&ired.  ^  ,  *  ' 
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mefine  the  characteristics  (e.g. ,  acf^, 
.  sex;  *race)  and  the'  variables  (e.g.', 
^employment  status,  earnings  ^ain)  on  * 
which  you  want  to, measure. 


A 
4 
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Determine  £he  precision-degree  .of  confidence 
fchatf  you  wan£  to  have  that-  the  sample 
findings  will .represent  the  true* popula- 
tion value  foremen  characteristic  or 
variable. 

For  example,  ify&nploydK  status  is  desired, 
ijs  it  satisfactory  to  hHjje  a  sample  finding 
whirh  is  plus  br  minus  5  percent  from  the 
true  value?    Or  is >plus  or  minus  15  percent 
gooa  enough?  ' 


•  Apply  the  appropriate  standard  statistical 
L  *     formula  for  estimating  the  sample  'size 

'  .     required  to  yield  the  findings  which  will  - 
be  acceptable  to  you. 

•  "Compare  resultant  sample  size  to  avail-  / 
able  resources. 

mf    if  necessary,  adjust  precision  and  conA  4flb 
>    ftfidence  leVels  eVen  further  to  obtain  a  w~ 

- <  sample -sire  commensurate— with  availatrte' 
^  *  resources.  -x  ' 


Control  and  Comparison  Groups 

'~        The "second  concern-  in  the  area  of  impact 
evaluation  is  the  use  of  control  or  comparison  groups.  The 
control  group  , is  a  necessary  element  in^ classical  experimental^ 
design  arid  impact  evaluation.     However,   it  is  not  a  possible 
of*  a  necessary  element  in  every  evaluation.     It  is  important 
tch^mphasize  t;hatmost  program  evaluations  do  not  require  a<  v/- 
controi/ group . 

TT^y&ur  PJC  feels  thafc  it  needs  to  know  the  afb solute 
difference  between  results  obtained  from  a  program  versus  no 

.  prograivaT  alj,  a  control  is  prob'ably  required.  -  In  most  of 
ypur  situations,  however,  the  real  choice  will  probably  be 
betweeh  one  form  of  service  ojpx  another,  lessening  the  ne"ed  for 
sjtxict  control  groups.     Two  options  which  your  PIC  might  employ 
as  basic  alternatives -to  the  control  gre,up'method  cfFe  cdmpa^ison 
cjroup  design  arid;  staged  design.       '  *• 


'  The  comparison  group  design  contrasts  a  comparison  group  m 
(available  individuals  who  have  not  participated  in  and/or.  - 
completed  a  program  or  project)  with  a  .similar  treatment  group 
(individuals  who  have  satisfactorily  completed  a  project).  ' 
Availability * and  similarity  are  the *two  key  elements  of  *this 
definition.     individuals  are  not  assigned  to  either' group  at 
random.     The  underlying  objective  of  the  comparison  group 
design  is  to#obtain>  by  self^-selection  rather -t^an  assign- 
ment,  as, much  ^similarity  as  pdssible  between^the  characterise 
tics  of  the  comparison  .group  and  the  treatrpelnt  g£oiip. 

-  Comparison  groups  which  have- been  used  in  various  man-* 
power  evaluations  have  fnclugled: 

m      Individuals  'who  register-  for  programs, 
but  are  subsequently  found  to  be 
ineligible     -  *    ^  (  • 

+      Individuals  who"  are  Found  eligible  for 
a  program^  but  who  leave  the  program 
before  assignment 'to  a  specific  project 
arid  without  receiving  participant  ser- 


< 


vices 


Individuals  who  activeljf participated 
.  "    in  a  program, but  who  terminated -prior 
to  completion  of  that  program 

Eligible  individuals  who  receive  sup- 
«  portive  services  only-  •  ' 


"  •    m  •     Individuals  registered  with  the  em-^ 
ployment  service,  but  who  are*  not 
*   -  enrolled  vith  CETA.  ■ 

T  -t* 
Availability"  is  the  principal  characteristic  of  ea<5h  of' 

these  possible  com^ari son'  groups.     It" is  obvious,  on  the 

surface^  that  e^bfc  of  those  group^  will  be  different  in 

some  way  from  the  progranT  enrol lee s .     Vet,  it  is  felt  -that  # 

their' similarities  maf  be  gteater  than' thei^y&if ferences 

and  thus  provide* the  basis  {or  valid  comparison. 
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One  technique  which  has  been  used  to  eliminate  as  many 
differjencls  as  possible  between  treatment  aijd  comparison  groups 
is  matching.     Using  this  technique;  the  evaluator  seeks  to 
match  individuals  in  the  treatment  group  with  individuals 
in  the  control  group  on  a  number  of  characteristics1 thought* 
to  influence  outcomes.     Doing  this  establishes  the  'basis <£or  \ 
a  more  total  contrasting  betweer^  directly  comparable  groups 
_Tand  thus  enhances  the  potenti^- co^fideftce  for*  the  findings 
which  result  ,fr'om  that  comparison.  ^ 

*  » 

>    2.     Staged  D.esign 

/  - 
^    The  staged  design  is  a  variation  of  th^^omparison  group 

approach  to  evaluation.     The  design  involves  Tsing  individuals  * 

who  are  registered  for  a  project,  but  *Jho  are  on  waiting  lists"  as 

controls  on^those  individuals  who  are  pro'ject  participants 

and -completers.  .  ) 

The  way  this  would  work  is  *ars  follows,:  *  Assume  that  a 
pool  If  two  hundred  eligible  participants " has  been  identified 
ahd  each  indicates  a  willingness  to  participate  in* a  PIC 
project  or  component.     However,  suppose  that  program  capacity 
is  limited  to  fifty  participants  per  cycle,*    You  could 
develop  a  staged ^design  by  jandomly  assigning  the  two  hundred 
eligibles  into  tRe  four\gr<5ups .     You  could  then  use  the 
persons' in  the;^fecond,  .and^  fourth  'cycles  as  controls 

for  individuals  to.  be  served/  in^hertirst  cycle.  The  third' 
and' fourth  group^  would  «erv\  ate  "the  controls  for  the  second 
cycle,  and'  so  forth. 

Tfce-s**^ed  designYapprotfch  avoids  the  denial-of-service 
problem, .employs  ec^i/albnt  groups,  and  uses  random  •assignment, 
It  also  has  som^ potential  disadvantages.  *  For  example, 

•  Since*,  tihe  assignment  is' random,  the 
, "representativeness "  of  each  group  • 
w>will  be  uncertain      •     _  • 

•  If  the  f^ogram  cycles  are  Short,  the  -  * 
post-meashrements  may  not, accurately 

"  -cte^ferience  or  charge  over  titne 


0 


•  The  size  of  later  treatment  *n<L  com- 
parison' groups  is  likely  to  diminisH 

/    as  individuals  find  suitable  employ- 
L    ment  or  tire  of  waiting  for  service, 
thus  the  equivalence  is  compromised 

•  If  .^comparison  group  individuals  inter - 
,  act  with  treatment  group  members  in 

the  course  of  their  noripal  activities 
in  the  community,  ^  sotirce  of 
 extraneous  influence  .enters,  


However,  in  spite  of  these  limitations,  there  appear  to  be  - 
many  situations  in  whivch  the  Staged  design  would  be# applicable. 


V 
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CONCLUSION 


In  conclusion f  collecting  evaluation  data  which  is 
•useful  demands  that  your  PIC  have  3  sound,  plan  and  research 
design*  good 'data  gathering  instruments,  and  skilled  pd^-r 
sronnel  who  can  implement  and  employ  those k instruments  \ 
according  to  their  design.  > 

-  :■  ■  I 

Manpower  Administration  (now  Employment  and 
Trailing ^Administration-  [ETA],-  Program 
Assessment  Guide   (Washington,  tD.C:  U.S.  * 
Department  of  Labor,  ^pril,  1974 f? 

*i  ^  , 

Mayors  Office  of  Manpower ,  -  Cifcy  of  Chicago, 
■Planning  and  Evaluation  under  CETA  (Washing- 
ton,  D.Cv:  U.^<^fepartment  of  Labor , v 
„     *  .  January  1976) . 

Olympus*  Research,  Self -Evaluation'  of  ,  CETA  Manpower  Programs:     ^  - 
A -Guide  for  Prime  Sponsors   (.Springfield,  .VA:  U.S.  Department 
of- Commerce,  1975).'       *  • 
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Analyze  Data 


Lys 


regate  and  review  *your 
ose  analytical  procedures 
bave  been  decided  upon  for 
is 'include:    cause  and 


"The  purpose  of  thi^step  is  to  . 
PXCs  evaluation  data  employing 
and  statistical  techniques  whi 
analysis-  -  Common  modes  of  a 

effect';  comparative;  trend;  ^and  cost  ^effectiveness 

Standard  techniques-  which  your  PIC  plight  employ  to  analyze 
d&ta  inglucle*    descriptive  statistics  and  correlation  \ 

■  /  .  '  ■ 


.  *  statistic. 


4 


3* 


■  \ 


Cetus^i  ycf  ;,Sf  Sectfr Analysis  study  of  |  number  of  variables  to 
'aecKSn^^atg  a  pa^sal  relationship  between  itwo  or  more  factors. 

'  fcomparativ'a  Ana\ysli>*,-?  .study  x>\  the  relative  performance  of 
tyo*  or  mo&e  -^otopSreht     -or  projects  overtime.    ,  ■  ) 

"Tre^d  An^lysi^  -  stvtay  b^j^jj^rTnaficfe  over  time  to  see  if  it 
,1-5  increasing,  decEea^Mkyr^or'  staying  the  sained 

Cost  Benefit  An'^lysia-  -  study  ^of  various  measures  of.  prS^ram 
>  outcome  related  tro  cdst  in  order  to  assess/ b£r\ef  its . 

.:  *  - 


•  * 


*  ■  V 


,^*#t  ((  toKwIed^/Capabimi^  Rati"?       j  '  '        .  ^  I'  '/ 

Re^l^W  the  irr^<5r^tipn  provided  above.  Ra'te  y?ur  Pic's  knowledge  and 
c^abili^ie^  with  reference  to  'thi^  step  on  the  scales* ^elt>w. . 

'  #  knowledge      •    'V>^  ill 


lent 


>  Capabilities 


Selow 


Poor 


Below 


Poof 


■  «     T.  '  .  f 

yptfr  ratings,  MecijSe  whether,  you  need  to  read'  the  content 
ion  on  thia  s^fe^iph  £>lK?ws  immediateJ^'or  prbceed  to  the 

-'''£■'•(?  *  *  /    .     *  .    •         -    v   '  7    .  ' 
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PME  PROCESS  NOTES/ ACTION  PLANNING  PAG]|_ 


NOTES  (Points  that  are  important  which  you  want  to  remember) 


s 


QUESTIONS   (Items  on  which  you  are  unclear  or  want  tx>  get  additional 
information)  -  " 


1  ACTIONS  (Specific 
1                    PIC's  PME 

1 

steps  that  you  intend  to  take  to 
pro£ess  in  this  area)  . 

implement  your 

1             . Steps 

»  Resources/Persons  **" 

Time  Frame  j 
*  • 

! 

j 

e 

,  4 
* 

•  f 

1              *  > 

r     -  Y 

< 

'  1 

 4  - 

»  '.  • 

•  * 

.4 

.            *                                              '  1 

*  1 

4  1 

S  • 
*  * 
• 

•  j 

erJc  ^ 
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'  INTRODUCTION 


Data  analysis,  or  data  treatment  as  it  is  sometimes 
called,  is  the> art  of  viewing  data  from  a  variety  of  per- 
spectives  in  order  to  understand  and  interpret  it.  Datar 
analysis  involves: 


 — reviewing  the  research  findings  for 

v  „  *  cofrelaticms,  trends,  commonalities, 
and  disparities  ) 

•       documenting 'the  relationship  between 
the  findings  and  the  evaluation  objec- 
tives '  _ 

-  .    .  .  A 

developing  ra^onales  or  speculations 
f<y  findings  vmase  explanations  are 
not„  self-evident. 

/  > 

«      *  .  •  \ 

Types  of  Analysis  • 

There  are  flour  basic  Jtypes  of  analysis  that  may  be 
useful  in  evaluating  your  PIC's  p^bgrams: 

Cause  and  Effect  Analysis:     weighs  all 
variables  to  demonstrate  a  caudal  rela- 
tionship betwefffTtwo  qt  raor^  factors.  4 

Comparative  Analysis:  focuses  on  relative 
'  achievements  of  two  or  more  groiips,  .units. 

♦   Trend  Analysis:    documents  chariges  in  * 
levels  of  achievement  over  time. 

^  Cpst-Benef it;     quantitative  ITidex  of  -  . 
performance  that,  relates  various  program 
results  to'cpst.  v  ? 

Thes^me£hods  have  been  described  earlier  in  this  Guide  * 

under  "St^ge  F:  Develop  Evaluation  Plan,  Step  1 

Establish  Scope  and  Nature  of  Evaluation, "  pp.III-259->14. 
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Analytical  Techniques'^^  ,  • 

The  analytical  techniques  your' PIC- will  employ  will 
be  dependent  upon  its  roode(s)  6T  analysis.     The  discussiort 
'  in  this^ectiojfrhighlights  two  of  the  more  common,  ar>3v 
simpler,  techniques  for  statistical  analysis.     These  are: 

•*     descriptive  statistics  *  ' 

"  ,  •      correlation  statistics?^ 

Other  analytical  techniques  frequently  used  by  evaluators 
include:    regression;  univariate  analysis  of  variance;  analy- 
sis of,  variance?  multivariate  analysis  of -variance;  and  chi 
square. 

Again,  as  with  the  other  bore  technical  evaluation  areas 
described  in  this  Guide,  if  your  PIC  decides, to  3pply*one 
or  more  of  the  above  i^pferenced  techniques,  you  shJuld  con«~ 
suit  d  statistics  manual  to  ensur<e  its  .proper  application. 
-  * 

1.     Descriptive  Statistics 

Descriptive  statistics  can  be  used  to  analyze  data  in^ 
te*ms  of : 

•        central  tendency  (i.e^,  mean,  median, 
or  mode) 

.   •       variability  (i.e.,  a. range  and  standard 
deviation) 

frequency  distribution  (i.e#,  the  num- 
v     ber  of  times  scores  are  achieved). 

-These  statistics  are  especially  helpful  in  looking  at. 
Overall  and  relative  performance  ;ln  projects  or  responses  to 
questionnaires.     For'example,  i£  your* PIC  ran  a  classroom 
ttaining  project  which  required  a  test' at  its ^conclusion, 'you 
could  rise  descriptive  statistics  to  compute  and  analyze: 

msap^  median,  and  modal  scores 

ahge  of  scores,  from  highest  to  '* 
|ovest  • 

r  v  •  r 
Btapdard  deviatipn  from  the  mean 

distribution  of  scores.  # 
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These  results  would  enable  you  to  c^i^riminate  individual 
and  group  scoring  achievement,  personally  and  relative  to 
others.     This  discrimination  would  allow  you  to  attribute 
effects  or  causation  to  t^e  project*  and  then  to -make 
decisions  based  upon  those  effects.  *'  • 


ft y 


2.     Correlation  Statistics 

'Correlaticjfri^ statistics*  are  used  to  measure  the  relation- 
ship between  two  variables:     a  dependent  variable  and  an 

^»  * 
independent  variable*    An  independent  variable  is  that  van- 

* 

able  which  is  manipulated  in  an  evaluation   (e.g.,  classroom 
'.training,  OJT,  a^e,  sex)  .     The/ dependent  variable  is  the 
factor  to'  be  observed  or  measured  to  determine  the  results" 
of  the  Variation  "(e.g. ,  salary  level,   length  of  job  r^en- 
tion,   self-concept  and  attitude)  . 

The  results  of  correlation  analysis  are  reported  within  - 
a  rajige  of  +1   (perfect  positive  correlation)  to  -1  (perfect 

* 

nfegative  correlation).     Correlation  statistics  c^n  be  used    ^  - 
to  measure  the  re£a^i©nshipj?etween"thfese ' two  variables,  but 
they  cannot  be  usjd  to  establish  causation.     For  example, 
your  PIC's  evaluation  may  find  a  positive  correlation  between 
classroom  training  and  salary  level,  but- it  cannot  prove  that 
one  is  the  consequence  of  tha  other.     Despite  this  li'mita-. 
tion,  correlations  are  extremely  useful  in  the  analysis  of 
your  PIC's  project  because  they  can  be  used  to  quantify  the 
nature  and  the  extent  of  relationships  between  independent 
variables     —  project  treatment  (i.e!,  services)  or  group 
characteristics   (i.e,,  demographic  data)        an^Jdependent  ; 
variables  -  results  achieved. 
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CONCLUSION 

r 

In  conclusion,  regardless  of  its  quantitative  aspects, 
data  analysis  is  muciWinore  than  a  mere  review  of  numbers.  It 
is,  rather,  a  systematic  attempt  to  derive  meaning  from  aftd 
make  informed  judgments  about  the  nature  of  the  program  being 
researched.  Regardless  of' the  form  of  analysis , your  PIC  is 
employing,  you  should  attempt  to  answer  the  following  basic 
questions: 


•  Werethe  PIC.1  s  or  the  project 1  s  objec- 
tives achieved? 

•  What  caused  the  effects  discovered  in 
the  evaluation?, 


Manpower  Administration  (now  Employment  and 
Training  Administration  '  [ETA] ) ,  Program 
Assessment  Guidrf  (Washington,  D.C.:  U.S. 
Department  of  J&bor,  April'  1$74). 


Mayor 1  s  0££it5e  of  Manpower,  City  of  Chicago, 
Plannin^and  Evaluation 'under  CETA  (Washing- 
ton, D.C.:  U.S.  Department  of  Labor, 
January  1976) . 

Olympus  Research,  Self -Evaluation  of  CETA  Manpower  Programs: 
A  Guide  for  Prime  Sponsors  (Spring-field,  VA:  U.S.  Department 
of  Commerce,  1975). 

Finks,  Arlene,  and  Kosec9ff,  Jacqueline,  Evaluation  grimer 
(Washington,  D.C.:  Capitor  Publications,  1978).  ' 
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Teedback  Into  Planning  Process 


The  purpose  of  this  slrep  ds  to  prepare  an  evaluation  report 
which  documents  your  PIC's  program  performance  for  the  year, 
The  report  isfed/back  into  the  Councils  planning  process 
and  then  usetPWr  re^lanning  for  the  following  year. 


•  'Evaluation  Report  -  document'  which  formally  summarizes 
and^presents  evaluation  findings.  \ 

•  Program  Review  Process  -  assessment  of  PIC's  programmatic 

•  performance  to  reevaluate  planning  assumptions  and, program* 
design  pripciples,  determine  project  and  service  deliverer 
effectiveness,  and  pinpoint  area  for  change. 


^4 


fl(  {{   Knowledge/Capabilities  Rating 

Review  the  information  provided  above.  '  Rate  your  PIC's  knowledge  and 
capabilities  with  reference  to  this  step  on  the  scales  below. 


•  Knowledge 

•  Capabilities 


1  

I 

1  .  , 

A  >» 

Excel- 

• Above 

Aver- 

Below 

Poor 

lent  * 

Avg. 

<  age 

Avg. 

1 

1 

 I  

i 

 I*i 

Excel- 

Above ,  . 

Aver- 

Below 

Poor 

lent 

Avg  .- 

age 

Avg. 

A 


Based  upon  your  ratings,*  decide  whether  you  need  .to  read  the  content 
'information  on  this  step 'which  follows  immediately  or  proceed  to  the_ 


next  step. 
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PME  PROCESS  NOTES/ ACTION  PLANNING  PAGE 


NOTES   (Points  that  are  important  which  you  want  to  remember) 


\ 


QUESTIONS    (Items  oil  which  you  ^e  unclear  or  want  to  get  additional 
information) 


ACTIONS.  (Specific  steps  that  you  intend  to  take  to-  implement  your 
*PIC's  PME  process*  in  this  area)  ~  • 


Steps 


Resources/Persons 


Time  Frame 


■  c 
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INTRODUCTION 


Once  your  PIC  has  analyzed  its  evaluation  data,  the 
last  tasks  in  the  evaluation  process  are:  ^ 

•        to  prepare  an  evaluation  report 

which  presents  the  results  of  that 
analysis  /  , 


to  .utilize  trhat  repbrt  to  begin 
the  review  of  program  performance  ■ 
ajid  to  plan  for  the  upcoming  year. 


A.  .  PREPARING  THE  EVALUATION  REPORT 


The  evaluation  te 


r  f 
mmar izes 

swer  those 

e  outset  of' 

to  derive  • 


umentl  whicl 

your  evaluation  fin'dingfs %     Your  feport  /houl<$: 
evaluation  questions  wfiidh  your  PIC  posed  aty> 
its  evaluatipn  and  explains  the  procedures  ujj 
the  answers 4  '  The  evaluation  report  is  the  t6£fici§il  record 
of  the  evaluation,  making  public  yolar  activities  and 
findings.     For  this  reason,  it  is  very  important  to  pre- 
pare the  report  ca^fully.  *  § 

The  degree  offformality  and  "sophistication 'of  -  your 
evaluation  report  ' lWxbviou sly  a  mattfc£/.of  local  preference. 
However,  a  basic  rule  is  that  your  r^j^rt  . should  neither 
consist  solely  of  statistical  tables  £>r  computer  printouts, 
nor  of  personal  impressions  or  opinions.     Your  report  shcAild  ' 
communicate  in  a  comprehensive  w^Yt^without  omitting  any 
qualitative  or  quantitative  detaiL^-what  has  been  discovered, 
what  was^one,  how  it  Was  done^^a^d  why^it.was  done. 
1        A  credible  evaluation  rep&xi,  clearly  afid  logically  \ 
describes  the  evaluation  questions  as  well  as. the  procedures* 
u§ed  to  ,get  the-  answers*.     Whether  informal  or  formal,  the 
report  should  incldfle  the  fallowing  sections: 
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Introduction  -  to  th£  program 'and  tHe 
evaluation*  » 


Objectives  -  of  the  evaluation. 


Methodology  -  means  designed  to  conduct 
the  evaluation . 

Findings  -  based  on  the  data'  collection  * 
and  analysis*.  »  ; 

Recommendations/Conclusions  -  based  on  , 
the  findings H 

Executive  Summary  -  briefly  explains 
*  objectives  of  evaluation  and  highlights 
"the'key  findings. 

These  sections  are  explained  in  a  little  more  detail  bel^ 
Much  of  the  information  for  the  first  three  sections  of  tj 
report  can  be  drawn  directly  from  your  PIC's  Evaluation 
Research  Design  (see  pp.   III-293  -  302).  - 

Introduction  S 


( 


The  introduction  to  your  Report  sllpuld  briefly  describe 
the  program  or  projects  being  evaluated,  the  grqup  that  con- 
ducted the  evaluation,  and  their  approach  to  evaluation. 
This  section  of  .the  report  should  discuss  stny  legislation 
that  created  the  program  and  mandated  its  evaluation.  It 
should  also  include  the  process  by  which  statements  of        :  * 
program  goals,  activities,  performance  standards,  and  objectives 
were  arrived  at  for  purposes  of  the  evaluatioh. 

Objectives  ^ 

This  section  should  state  succinctly  the  specific  6bjec- 
tives  for  the  evaluation  and  list  the  question^ ♦ that  were 
to  be  answered  through  the  evaluation.  " 

Methodology  *  ,  ^ 

TRi^  section  should  present* your  overall  evaluation  — 
* 

research  design,  outline  the  process  for  administration  of 
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the  evaluation,  describe  data  collection, instruments/and 
procedures/  and  youy  PIC1  s  analytic  t^hniques  andkaata 
analysis  results. 

Research  Design .     If  your  PJC  concentrated  upon  per-  n 
fo^mance  versus  plan,  or  relative  effectiveness  analysis,        }  * 
that  strategy  should        described  here.  »  If  yoOr  extended  intcjj  ,  . 
impact  evaluation,  yoii  should  describe  that  strategy Jalso  and 
elaborate  on  your  sampling  procedures  and  any  limitations  on;] 1 
them.     For  each  evaluation  of  a^  impact  nature,  you  should- e#- 
*plain  and  justify  the  way  participants  or  employers  were  ; 
grouped,   t>he  independent  and  depenSent  variables,   the  sample^^ 
and  any  limitations  kwi  ^internal  and  external  validity.     A  *  , 
drawing  of  the  d wkjgm  is  also  very  helpful. 

The  discu5^B^Bf  ti)e  plan  used  to  select  participants  q£ 
employers  for  .tty^Mfituation  sholi^l  include  the  Sampling  pro* 
cedure  used  fc  (e.  g.J^^urposive  sampling);   the  justification  far 
any  subdivisions  pr  strata  'inter  which  the  population  of  poten- 
tial participants  was 'divided  forfc  sampling  purposes;  the  final* 
numbers  of  individuals  in  the  sample;  and/any  problems  en- 
countered in  selecting  the  participants  oi 
are'  inherent  in  the  sampling  plan.4   Once  ag^in,  giving  a 
picture  of  the  final  sample  is  useful.     FinallyT^^it  is  important 
to  discuss  how  well  the  procedures  produced  the  desired  sample, 
at  ieast  insofar  as  can  be  ascertained  through  statistical 
method^  of  comparisons  with  tables  of  demographic  data., 

Asministration.     This  subsection  describes- the  actual 
steps  taken  to. implement  your  research  design.     It  provides 
information  about  the  sequence  of  events  between  defining  your 
evaluation's  quest iQns  and  arriving  at  the  answers.  Informa- 
tion .about  the  staff's  activities  may  be  appropriate  in  this 
section  of  the  report.     One  way  to  combine  information  about 
the  evaluation  schedule  and  staff  is  to  draw  up  a  calendar  * 

of  events  describing  who  did  what  when  in  the  evaluation 

?  '  - 

'process.      '  *  * 


limitations  tfiat 


.Instruments 'and  Data  Collection  Procedures.     This  sub- 
jection should  describe  instruments  employee!  in  your.PIC'-s 
evaluation.     Each  instrument  and  the  people  to  whom,  it  was 
administered  and  data  collected  'by  it-*should  be 
detailed The  entire  instrument  should  be  reprinted  if  * 
^possible,  and  if* not, *  sample  items  should  be  given. 

-    °Thi€  subsection  should  also,  contain  information  about 
•the^  reliabilit,y^*id  validity  of  the  instruments Were  they 
pilot-teste$  or ^validated?    If  so,  with  and  by  whom?  What 
were  the  results?    It  is  also  a  good  idea  to  explain  how 
the  instrument  was  .administered  and  scored.  '  9 

If  personal*  contact  was  part  of  your  evaluation  .i&ode,  - 

•         '        *  0 
then  air  field  activities  should  be  summarized  here,  noting 

any  irregularities  in  information  collection  that  could 
affect  the  evaluation's  findings.     If  interviewers -strayed 
from  the  set- procedures  for  interviewing,' for  example,  then 
the  report  shguld  describe  what  happened  and  what  the  con- 
sequences werev     Finally,  this  subsection  should  report  he 
many  jpeople  participated  fin  each  information  collection 
activity,  for  how  many  of  them  complete  data  werp  available, 
and  give  reasons  for  any  missing  information. 

Data  Analysis  and^  Treatment  >  i  Eac^i  clata  analysis  method 
used  by  your^  PIC  should  be  named  apd"  described  in  this  subr  ' 
section'.     For  each  separate  analytic  method",  a 'description 
should  be  given  of  its  relationship -to  the^pvaluation  ques- 
tions, the  sousde  of  information  for  the  analysis,  the  design 
strategy  (including  tttte  independent  and  dependent  variables) 
and  any  limitations.     Also  the  analysis  results  and  inter- 
pretations of  the  findings  that  delate  tfrera  to  the  evaluation 
questions  should  be  given  for  eachVaTtalytic  procedure   you*  - 
employed.'        '  ,  ^  ♦ 

Findings'  '  :  .  * 

The  most  cri'tical^part  of  'your  PIC's  evaluation  report 
is  the. answers  to  the  evaluation  questions.     The  report  must 
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provide  qlear  and  succinct  answers  pr  describe  the  progress 
being  made  tpward  obtaining  theta.    When  reporting  answers  * 
to  evaluation  questions,  it  is  important  to  point  out  the 
strengths  and  weaknesses  of  the  project^ or  program.  ^This 
becomes  important  in  later  decision-making  and  program 
modification  sessions. 

Recommepdat  ion  s/Conc3rus  ions 

This  section  presents  ma jor Conclusions  wHigh  can  be 
drawn  as  a  result  of  answering  all  the  evaluation  questions  § 
which  were  a # focus  for  this  year's  inquiry.  Recommendations 
4  sortte times  accompany  conclusions  or  answers  to  questions.  You 
*should  determine  in  advance  if  recorpmendations  are  required, 
and  if  they  are,  how  extensive  they,  shoul4  be.     If  the 
decision  is  to  include  recommendations,  the  report  should 
not  assume  the  Council's  responsibility^ of  deciding  whether 
pr  not  to  continue  a  project's  funding.     Instead,  this 
section  should  present  recommendations  oo  how  to  improve 
the  project,  on  the 'situations  in  which  the  project  can  > achieve 
*the  best  results,  and  on  the  individuals  who «are' most  likely 
to  benefit  from  the  project.     In  addition,  this  section  might 
als©  present  other  program  options  which  the  .Council  might 
consider  as  alternatives  in  .Preplanning  for  the  next  fiscal  year 

In  answering  the  evaluation  questions,  drawing  conclu- 
sions,  and  in  making  recommendations,  it  is  important  to  ex- * 
plain  the  limitations  imposed  upon  your  evaluation.  For; 
example,  'if  information  came  from  instruments,  the  reliability 
or  validity  of  which  is  uncertain,  the  reader  should  be 
reminded  about  it  when  the  evaluation  question  is  apswered. 
Your  report  must  pinpoint  those  limitations  on  the  evaluation 
findings  even  if  this  makes  the  report  appear  noncommittal 
fand  the  results  somewhat  more  difficult  to  translate  into  ■ 
policy  or  action.  m 

■    ■  )  '-■ 

Executive  Summary  , 

section  takes  the  activities  and  findings  de^cjibed 
i^the  body  of  your  PIC's  evaluation  -report  vand  distills  'them 
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'into  &  few  pages.     The  purpose  of  the'  Summary  is  to  give 
people  an  pverview  of  the  evaluation  that  is  easy  to  read 
£ut  detailed  enough  to"~be  believable  and  usable.     The  summary 
is  frequently  placed  at  the  Beginning,  of  an  'evaluation  report 
as  a  special  overview.^  It  is  h.  convenient  devifee  for  informing 
.the  public  about'  the  program  and' its  evaluation.     Because  it 
may  be  more  widely  distributed  than  the  complete  report,  the 
•evaluation  summary  must  be  carefully  prepared.*    A  good  evalua- 
^tion  summary  must- be  understandable  and  convincing  for  all 
readers  regardless  of  their  sophistication  and  personal 
motivation  to  read  findings  regarding  the  program. 


B.     BEGINNING  THE  PROGRAM  ^REVlW  PROCESS 

Your  PIC's  evaluation  report  is  'ffte  primary  \documeryt 

for  program  review.  •  It  should  be  fed  back  into  your  Council ''s 

f 

planning  process  and  then  used  as  a  primary  tool  to: 

•        reassess  planning  assumptions  $nd  pro- 
gram design  principles  „ 

00     •        determine  'project  and  service  deliverer 
effectiveness 

t  *  * 

V       pinpoint  a*reas  for  <  change. 

"?    This^process  is  described  at  lejfgth  under  "Stage  B:  Define 
PIC  Strategy,  Step,s  1-6, "  pp. 111-6/  -  136-     To  summarize,  your 
evaluation  results  can  help^you: 

*   •  *     to  determine  whether  major  or  minor 
changes  are  required  in  your  PIC's 
prograjn  scope,  approach,  and- project 

•»    \  mix 

«* 

<•        to  provide  a  perspective  from  which  4 
to  assess  the  appropriateness  of  your 
pfyrformance  standards  1 
% 

* '  ±  •        to  permit  comparisons  among  components 
*and  projects  to  ascertain  which  are 
most  effective  and  cost-efficient  •/ 

;    •        to  decide  whether  particular  projects  -  J? 

*   •  should  be  continued,  changed,  or 

■  •    ,  terminated.  \ 
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In  conclusion,  your  evaluation  findings  provide  the 

**  •  •  •  • 

necessary  information  which  your  PIC  can  employ  to  develop 
an  improved -program  operations  plan  for  the  next, year. 


Mayors  Office  of  Manpower,  Cit^y  of  Chicago 
Planning  and  Evaluation  under  CETA  (Washing 
ton,  D.C.:  tf.S.  Department  of  Labor, 
January  1976)  . 
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IV.     PME  AIDS 


\ 


This  section  contains  draft  sample  questionnaires, 
forms,  and  worksheets  which^  you  can:  use  as  protptypes 

.ancji  referehces 'to  develop  your  own  tools  for .  implementing 
the  PME  pxocess.     These  aids  have  been  created  primarily 
employing,  or  based  upton*  examples  and  information  pre-  € 
sented  in  Stection  III  of  this  Guide.        *  t* 

The  aids  are  organized  by  stage  and  step  as  outlined 
in  that  Sectipn..  The  a id &  for  each  stage  a*e  introduced 
by  a  E^age  which  lists  them  by  step  and  jrefers,  you  back  to 

*  tiiose  pages  in  Section  III  :which  provide  content  informaf- 
tion  related % to  ,the  step.     There  is  at  least  one  aid  for 

'every  step  in  the  PME  process.  **  Each,  aid  is  presented  with 
a  One-sentence  explanation  of  its^us-e.  %  „ 

It 'is  anticipated  that  more  specific  model  aids  f or m 
implementing  the^PME  process  will  be  developed  as'  a  result 
of -the  field  test  with  the  £en  selected  pVlot  PIC<>. 
Once  6hese  aids  are  developed,  .they  wi\l  be  forwarded  to 
you  to  replace  or  supplement  the  material  presently  housed 
.in  this  section  of  the  Guide. 
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A.   ASSESS  LOCAL*NEEDS  * 

•  ,  ,  i 


pMe  aids 


Step 

1.  Analyze  Local  Labor  Market  Conditions 
and  -Business  Needs    ,       -  ^ 

1.1  Labor  ri&tkefc  Analysis  Questions 

1.2  Erie-initiation  Local  Employer 
^>  Survey  Questions 

1.3  Local  Employer  Survey  Questions 

1.4  '  Labor  Market'  Analysis  Format 

1.5  Labor  Market  Analysis' Projection 
Format 

2.  Analyse  Loyal  Population  Groups  yind 
Service  Needs 

.  2.1    Master  Plan  Review  Questions 

2.2  Significant  Segments  (CETA- 
Eligible  Participants)  -Worksheet 

2.3  Significant  Segments  Percentage 
Analysis  Format  *       *  > 

\2-4     Significant  Segments  Cross 
Tabulation  Format* 

3.  Review  Programs  and  P^ans  of  Other 
Agehcies/Sferyice  Deliverers/Businesses 


Section  III 
Reference  Pages 


III-13-^l 

111-27 
111-28 
111-31-34 

111-35 


111-45-46 


111-47 


II*-47 
111-49 


3.1    Plan  and  Program  Review  Questions  111-56-59 
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".1     LABOR  MARKET  Ag^LYBXg  QUESTIONS- 


USE:    .To  determine  basic  characteristics  and  structure  of  the  total 
labor  market.    . ,  *    .  /* 


QUESTIONS* 


T^^^Ka^^^^Ji^^eogralplS 


Ti     What  is  "the  industrial  composition  of  ths-  ?rea? 


What  are  the  employment 


^levels  ^rn 


ong  industries? 


4.     Who  are  the  spec 

V 


i  f  ic  ei^? 


loyers  in  industries? - 


5.     What:  are/have  b*n  the  historical  employment  trends  in  the  area 

over  the  past  5  years?  .  /  H 


I  ■ 
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o^^Wna^x^^ne^ctirren^Tc^ 


What  does  other  data,  such  as  turnover  and  job  openings,  indicate 
about  recent- ind lis tri&l  trends?  •  v*    ,  '  / 


1H    What  are  industry  employment  projections?  \ 


/What  are  the  employment* potentials  of  lopal* industries? 


[0~I    Who  are  the  local  employers  in  *  high  Employment  potential 


industries?" 


■  '490  •  \  •  , 


lat  -is  the*  opcupationaJ 


12./  What  are  occupational  projections  within  these  industries? 


•  V 


13. 


What  is  the  quality  of  available  jobs? 


f  -  >  > 
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■_^__jL±2     PRE^INITIATION  LOCAL  EMPLOYER  SUEVEY  QUESTIONS 
ySE;     To  decide ^whether  to  conduct  a  local  employer  survey.'  . 


/   •'  QUESTIONS' 

*  •  *  • 

1.       What  is  yopr  'information  'neeTd?  * 

•  •  /  What  data  are  needed?  / 

/  How  will  it-b«  used  in  the  planning  process? 
.  Will  additional  da,ta  he>lp4you-  make  better* 
decisions?   •  ✓ 


2.       Can  you  .utilize  existing  Resources? 


Is  data  available  from  traditional  resources  such  as 
State  Eiqgloynrent  Security'.Agency  (SESA)   and  State  ■ 
Occupational  Inf ormation%Coordinating  Committee  (SOICCf? 


3*      Is  the  survey  useful  for  your  PIC?  •? 

.  Is  it  cost  effective^ 
'    •  Does  it  employ  correct  survey  procedures? 
.  Will  the  data  it  gathers  be  reliable?  * 
-    "     .  Are-  there  alternatives  to  the  'survey? 


Will  the  investment  in  the  survey  be  worth  the  results? 

,o  •  *  + 

*  *  «  ^ 

.  "Can*  your  total  information  requirements  be  'reduced? 

.  Can  alternative  data  be  used?.  .  * 

.  Can  any  data  needs*  be  dropped  without  'adversely 

impacting' the  planning  process?  .   .  , 


n 
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1.3     LOCAL  EMPLOYER  SURVEY  QUESTIONS 


"USE:     To""%&ther  in-depth  information  regarding  the  nature  of  "the 
locaX^abor  market,  job  opportunities,  and  employer  needs. 
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■  /  •  , 

-6.   ^  What  are  thfe  number  and  occupational  characteristics  of 
^  existing  j<fb  vacancies  in  loc^l  labor  markets?^ 


^fhat *jfre  the  hiring  practices  and  policies  of  local  firms? 

$  ■ 


i 


• 8^      Whaft  are  the  hourly,  weekly  and/or  annual  salaries  of  workers 
in  specific  occupations?  *.  •   *  , 


9.      What  are  the  training  and  promotional  policies  of  local  firms? 


10^    ]  What  are  employer  attitudes  toward  CETA  programs? 
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1 .  4     iA^R  MARKET  ANALYSIS  FORMAT 

fro  ^identify  pr: 
k>cal  economy. 


> 


USE:    >o ^identify  private' sector  job  opportunities  within  the 


i  workers  fc 
available 


Private  Sector 

Textile 
(Synthetic) 

warehouseman 
dyer* 

loom  operator 
etc. 


Interstate 
Trucking 

machinist-shop 

refrigeration 

mechanic 
diesel  mechanic 
driver 
etc. 


Canning^ Frozen 
good  Processing 

laboratory 

technician 
picker 

plant  mechanic 
machine .tender 
etc.    .  : 


Banking  , 
teller 

computer  prog 
4latat  processing 

clerical 
account  clerk 


\ 


3 


Current  demand 
for  workers 


Estimated  future 
demand  (give 
date  for  - 
projection) 


Estl 
supp 
for 


eT^uturs 
date 

onK: 


lmaJed  fut 

pi/  (give 
rtojectlo 


Non-CETA  training 
currently  available? 
If  yes.  Indicate  how 
ny. workers  will  be 
trained 


49o] 


1*.  5-  LABOR  MARKET  ANALYSIS  PROJECTION  FORMAT 


USE: 


To  projecjt  potential  future*  job  openings  within  local 
businesses  and  industries. 


 7  

t 

EMPLOYMENT 

ANNUAL  AVE  RAG 

E  JOB  6PEN 

INGS 

Occupational  Titla 

1*74 

1986 

Changs 

1974-1986 

r 

.  L  j  

Job  Opening. 

Actual 

Projactsd s 

Numarical 

Par  cant 

E  xoansion 

Raplacarnaot 

Totdl 

Rdlio 

Crafts  and- kindred  worfcart 

19.133 

23.173 

4.040 

21  1 

367 

464 

-  - 
631 

~~          "  ~~ 
4  3 

Construction  crafts  4 
*  Carpantars  and  appranocaa 

-6.116 

6.473 

1.368 

i  26  8  • 

123 

125 

4  6 

1.689 

1,903 

'314 

198 

29 

42 

n 

4  b 

8  nek  and  stonamssons 

f 

and  appranticac 

316 

370 

66 

176  « 

6 

6 

1 1 

3  b  * 

BoMdoiar  opar  ators 

106 

176 

67 

82  0 

6 

2 

6 

7  4 

Camant  k  concrata  tVasnars 

•  V78 

238 

60 

*    33  7 

5 

3 

4  5# 

^Wctnctans  k  appranticaa 

863 

1.180 

327 

38  3 

30  . 

18 

46 

Excavating,  grading. 

mach  oprs 

230 

396 

166 

72  2 

16 

6 

21  y 

9  1 

Floor  layars.  axe  t*a  aattart 

33 

36 

2 

6  1 

0 

1 

1 

30 

J      Pamtar •  and  appranocaa 

733 

749 

16 

2  2 

1 

21 

*' , 

30 

Paparhangart 

46 

*1 

13 

.  27  1 

1* 

3 

4 

6  3 

Ptastarars  and  appranticaa 

22 

26 

0 

13  6 

;  0 

1 

1 

4  5 

Pkimbars  ptpaftttars  k  appran 

686  ' 

941 

263 

36  8 

.  i  23 

.17*  . 

40  * 

5  8' 

,  Jtoofar  t  and  afatars 

89 

123. 

34 

38  2 

•  \ 

1  1 

4  4 

4  5    1  ■ 

Structural  mat  at  c/*ftwork#*  t 

180 

216 

39 

20  0 

'  3 

3 

b 

'  33 

Taaaat|art 

49  * 

61 

12 

24  6 

i  • 

2 

4  1 

2.770 

,3:630' 

760' 

27  4 

69 

64 

1 11 
1  j  j 

4  6'  • 

Matafwortung  crafts'. 

V 

I 

axe  machamcs 

.  U*69 

1722 

263 

<  180 

|  24 

32 

56 

,  3-8 

ilacksmtths 

13 

9 

•308 

'  0 

1 

1 

49 

•oaarmakart 

36 

48 

13 

37  1 

1 

1 

2 

62 

Haat  traatars.  annaalars,  ate 

27 

26 

1 

37  . 

0 

1 

1A 

40 

Forga  k  hsmmmf  opar  at  or  t 

31 

4 

14  8  * 

0 

1 

1 

SI  , 

Job  k  d«a  aattart.  maul 

^  126 

129 

3 

,2  4 

0 

*  2 

2 

16  " 

Machinists  k  appr  ant  teas 

496 

676 

81 

184 

•7 

1 1 

16 

3  7 

Miawrights 

'MoWars.  matal  k  appranucat 

60 

113  h 

33  . 

41  3 

3 

2 

5 

63 

46 

46 

•1  ( 

2  2 

0 

1 

) 

22 

Pat  tarn  k  modal  makars 

29N 

i?" 

6 

•20  7 

1 

1 

'  -O, 

is 

HoNart  k  hmshart,  matal 

17 

4 

23  6 

.  0 

0, 

•  0 

Snaat  matal  workart  k  appran* 

326 

C  366 

47 

20  4 

•  6 

6 

*  12 

3  7 

Tool,  dta  makart  k  sppr arnicas 

236 

304 

68 

28  8 

8 

6 

1  1 

■      0  c 

4  7 
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Is  the  d&ta  timely? 


u 


Do 'the  sources  of  data  and  their  time  periods  xeflect 
current  conditions  in^your  area? 


Is  the  dteta  comprehensive? , 

.  Was  more  than  one  standard  source  used  to  analyze  and 
define  population  groups  and  their  needs? 

ted  data  used  to  supplement  existing 


.  Was  locally-genera 


X  Doli  Masfer  Plan  reflect  analytical  insights? 

.  Arte  percentage' relationships  used? 

.  'Are  c^oss-tabulat£ons6f  data  used?  , 

0  Are  other  ptatist^cal  measures  used  to  gain  Analytical 

^       insights?  t 


4. 


Has  the  Master  Plan  assessed  the 'needs  ofe  "special;  population 
groups  such  <as: 


^-disabled  and  Vietnam-era  veterans 
— public  assistance  recipients    A  ' 
--older  workers  (55  years  of  age  or  older) 

— ex-offenders  •  - 

I 

— displaced .homeraakers  *  — 

— handicapped  persons 

7-single  parents  *  r 


-individuals  who  lack  credential-s  (auch  as  persons 
lacking  licenses,  required  experience,  etc.") 

-individuals  who  require  basic  and  remedial  skills 
development 

-alcoholics  * 
-migrant  workers. 


5. 


Does  the  Master  Plan  explain  tl^e  cause  of  participant 
group  needs  such  as: 

— education  level 

—  "fcransporitation  [  \ 

— language 
— child  core 

—world  Qirwork/employabiiity  orientation 
4  .    — skill  development. 


ERJC 
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-USE: 


*>     2.2     CETA-ELIGIBLE  .PARTICIPANTS  WORKSHEET  V 

To  record'  numbers  of  CETA-eligible  participants 
by  segments' (i.e. ,  age,  race,  sex,  ethni^ty  of 
local  population) .  +  /  ' 


9 
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♦  * r*  *        .  y  -  * 

\  2.3     PERCENT  OF  CETA-feLIGIBLE  PARTICIPANTS  BY 
SIGNIFICANT  SEGMENTS  ANALYSIS  FORMAT 

*  V  •  *  * 

USE:     TO  REVIEW  REJ^ATIVfi,  PROPORTIONS  OF  SIGNIFICANT 

SEGMENTS  OF  LOCAL  POPULATION  WHO  ARE  CETA-ELIGIBLE  ^ 


# 

General 
Population 

Estimated 
number  of 
persons 
potentially 
eligible'  for 
CETA  prbcprams 

Percent 
of  CETA- 
eligible 
partici- 
pants 

Total,  For  PS  Area*  „ 

DEMOGRAPHIC  GROUPS 

Sex 

tfjale 

 2  3  w 

'  Female  \ 

19  &  Under 

* 

 :  

* 

<20>-  21 

- 

Age 

22  -  44 

45  -  54  ^ 

- 

55  and  over 

'White',  (not 
Hispartic 

 :  A  

Race/  • 
Ethnic 

Black  ^4#ot  1 

Hispanic. 

 :  T+  ■  

Group 

Hispanic 

Y 

American  Indian 
or 

Alaskan  .Native  ' 

r 

I 

f  , 

Asian  lor  Pacific  « 
Islander    ^  >* 



-  * 

• 

•  ,  500    .  - 

ERIC  . 


7  2  4     PERCENTAGE  OF  CET A-ELIGIBLE  PARTICIPANTS  BY 

Significant  segments  and  category  format 

USE:    To  conduct  focused  analysis  of  population  groups 

  employing  two  or  more  f actors- je. g. ,  age  and  sex) 

in  conjunction. 


Male 

• 

-19  &  Urider 

;     /  . 

General 
Population  - 
Total  A  

Number  Eligible 

%  CETA- 
Eligible 

White  (not 
Hispanic) 

f Black  (not 
"  Hispanic) 

* 

Hispanic 

• 

American  Indian _ 
or  Alaskan  Native 

*  ■ — - 

Asian  or  Pacific 
Islander  . 

r 

*  •» 

1 

9 
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\           3.1     PLAN  AND  PROGRAM  REVIEW  QUESTIONS* 
 '  *    

USE!:     To  analyze  .plans  and"  programs  of  others  to  identify,  points 
for  collaboration,  service  gaps,  and/or  models  to  replicate, 


'  QUESTIONS 

,  Review  the.  plans  of: 

(  J      the  CETA  Prime  Sponsor 

'  Employment  Service  j 

7*  '  *  • 

(    .       educational/vocational  courses  run  by  local  high 

schools,*  community  colleges,  or  proprietary  insti- 
*      m      tutions  • 

/   .       economic  development  and  other  employment  and  / 
'training  initiatives  sponsored  through  the       *  9 
federal  government  or*  by  the  local*  government 

employment  and  training  efforts  undertaken 
by  private  sector  firms. 

To  determine:  \ 

,   .       Are  there  gaps  in  the  local  service  delivery  system? 

Are  there  possible  areas  for  collaboration  with  service 
•  deliverers? 

; 

Are  there  a/eas  in  which  your  PIC  may  choose  to 
.initiate  new* services  or  projects?  * 

Can  you  "piggy  back"  by  using  PIC  funds  to  support 
successful  programs?  < 

Are  there  existing  program  designs  that  you  can 
-modify  and  Use  in  your  PIC? 


4 


B.   DEFINE > I C  STRATEGY 


1. 


6. 


PME  AIDS 


Section  /III 
Reference  Pages 


Assess  PIC  Program  Performance 

1  » 

1.1  Program-Level  Assessment  'Formats 
— Service  to  target  groups 
--Enrollment  and  expenditures 

•  — Terminations 
— Costs 

— Other  services  and  activities 
— EGS  activities, 

1.2  Component-Level  Assessment  Formats 
— Costs 

—Enrollment  and  expenditures 
— Terminations 

1.3  Project  or  Service  Deliverer^Level 
Assessment  Format* 

1.4  Program  Assestfnent  Questions  ' 

Assess  PIC  Organizational  and  Operational 
Performance  I 


111-62 


III-7C 

'IlV-81- 
111-83 


2.1    Assessment  Questions 

2.2-  Developmental  Need  Questions 

-Establish  initial  Need  Priorities 


111-87-88 
111-88-89 


r 


3.1  EGS  Needs  Questions 

3.2  Target  Group  Selection  Questions 

3.3  Training  Program  Occupations 
Selection  Questions 

3.4  Target  Group  Service  Needs  Format 

3.5  Target  Group  Assistancfe  Needs  Matrix 

Redefine  PIC  Mission  and  Function 

4.1    Mission  and  Function  Analysis  Form 

Refine  Basic •Organizational  and  Operational 
Framework 

« 

5.1  structural  Analysis  Checklist 
Formulate  Goals  and  StraWifc  Plan 
6. 1    Go*i  Setting  Form  


111-93-96 
111-96-99- 

III-99t101 

III-101-103 

III-101-103 


\ 


111^07-115 

III-119-127 
III-131-136. 
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SEX 


•RACE 


AGE 


1.1  »  PROG RAM -LEVEL  ASSESSMENT  FORMATS 

USE:    To  review  program-level  performance  in  order  to 

  ascertain  overall  PIC  programmatic  *ueo*s  .and 

'  •     to  make  fundamental  decisions  regarding  program, 
direction.  ;         \  '   -  . 


PERFORMANCE  VS.  PLAN:    SERVICE  TO  TARGET  GROUPS 
X  of  Total  Participants 


GROUP 


ACTUAL 


PLAN 


%  -VARIANCE 


SPECIAL  ' 
CATEGORIES 
(as  defined 
by  PIC)  ' 


Male 
FemaVe 

White/Mon- 
Hispanic 

Bl ack/Non- 
Hispanic  • 

Aifier .  Indian  or 
Alaskan  Native 

Asian  or  Pacific 
Islander 

19  and  under 
20-21  - 
22-44 
h  45-54 
55-older 


Handicapped 

Vietnam  Era 
Veterans 

ADC  Recipients 

Ex-Offenders 

Others  (list) 


IV- 21 
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PERFORMANCE  VS. -PLAN:    ENROLLMENT  AND' EXPENDITURES* 


INDICATOR 


ACTUAL 


RESULTS 


PLAN 


VARIANCE 


Total  CumuVative  Enrollment 
Total  Current' Enrollment- 
Total  Accrued  Expenditures 


PERFORMANCE"  VS.  PLAN;  TERMINATIONS 


-A 


NDICATOR  , 


i  Positive  Termi nation 
Rate 

.  Entered  Employment 
Rate 


Indirect  Placement 
Rate 


Private  Sector  '  W 
PtUcement 


FORMULA. 


Tot.  Pos.  Terms.  ,nn 
Total  Terms. — 5  *  100 


Tot.  Ent.  Emp. 
Tot.  Terjns.  * 
Less  Transfers 

Tot. Indir. Place. 
Tot.  Terms.. 
Less  Transfers 

Prty. Sec. Place. 
Tot.  Entered 
Employment 


x  100 


x  100 


x  100 


ACTUAL 


-RESULTS 


PLAN  . 


%  VARIANCE- 


IV- 2  2 


Son 


— .  *  -             •         ..    •  

'     ^               PERFORMANCE  VS.  PLAN:  COSTS 

\         1    ■ 

1   

'  INDICATOR 

t  ■ 

ACTUAL 

PLAN 

%"  VARIANCE 

  * 

•  Cost  Per  Positive 
Termination 

.  Cost  Per  Entered  -  , 
EmDlovfnent 

•  Cpst  Per  Indirect^  / 

•  ^Placement 

Tot.  Accrued  Exp. 

— — —  ■* 

« 

 ■  . 

« 

% » 

Tot.  Pos.  Terms. 
Tot.  Accrued  Exp. 

Tot.  Entered  Emp. 
Tot.  Accrued  Exp, 

Tot.  Indir.  Place. 

Of 


IV- 2  3 
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-    PERFORMANCE  VS.  PUN:  VTHER  SERVICES  AND  ACTIVITIES 

 i—L  — * — 

SERVICES  AND  ACTIVITIES 

KfcMJLIS 

ACCRUED 
.  EXPEND. 

$$  BUDGETED 

V 

ACTUAL 

PLAN 

%  VARIANCE 

Administrative  Cost  Pool 
Intake  and  Assessment 
Employment  Genera tUg  Servs. 
Other 

*  * 

i 

IV- 2  4 
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PERFORMANCE  VS.  PLAN:    SUMMARY  -REVIEW  FORM 


OBJECTIVES/ 
COMPONENTS 


ACTIVITIES 


NOV  RESULTS,  IN- 
COMES. PRODS. EXP. 


ACHIEVED 


YES 


NO" 


ASSESSMENT 
QUANT.  I    QUAL.    1  .TIME    1  COST 


perf; 

LEVEL" 


£XPLANATION 


\ 


H 
< 
I 

Ul 


9 
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USE: 


1.2     COMPONENT -LEVEL  ASSESSMENT  FORMATS 

4  —  % 

To  analyze  and  compare  the  performance  of  the  various 
components  constituting  the  PIC's  program. 


C 


PERFORMANCE  VS.  PLAN:  COSTS 


< 
i 

to 


T 


COMPONENTS 


Classroom  Tjjftg.  Occ. 


Classroom  Trng.  -  Other 


OJT 


Upgrading 


Retraining 


E6S 


"Other 


Other 


COST  PER  POSITIVE *TERM* 

.    COST  PER  ENTERED  EMP.* 

i             CQST  Pl 

ER  PARTIcfpft* 

-Actual 

Planned 

%  Var. 

Actual 

Planned 

%  Var.': 

'  Actual  • 

PUnned 

^varTif 

« 

r  ' 

— fc— 

> 

 1  

< 

• 

i 

'    -    -  -1 — 

SeVect  1n'd1< 


ERLC 


*Se*Iect  Indicators,  based  upon  the  nature  of  your  program,- which  are  applicable  to  components. 

.  510 
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PERFORMANCE  Vs/PLAtiP  ENROLLMENT  AND  EXPENDITURES 


M 
< 
I 


COMPONENTS 


Classroom  Jrng/  *  Occ. 


Classroom  Trng.  Other 


OJT 


Upgrading 


Retraining 


E6S 


Other 


Other 


COMPONENTS' 


Classroom  Trng.  -  Occ. 


^Classroom  Trng.  -  Other 


OJT 


Upgrading 


Retraining 


E6S 


Other 


Other 


CUMULATIVE  ENROLLMENT 

CURRENT  ENROLLMENT  . 

ACCRUED  EXPENDITURES 

Actual 

Planned 

*  Var/- 

Actual 

Planned 

X  Var. 

Afitual 

Manned 

X  Var. 

• 

f 

.3 

* 

» 

r 

• 

^PERFORMANCE  VS.  PLAN:  TERMINATIONS 


POSITIVE  TERM  RATE* 

ENT 

ERED  EMP.  RATE* 

INDIRECl 

<  ^  

'  PlApFMFNT  RATE* 

Actual 

Planned 

X  Var. 

Actual 

Planned 

Actual 

Planned 

C.Var 

— - — a — 

5 

— r1  

 r  

>^—  

*  * 

• 

 ^  

7  

m 

• 

4» 

^Select  indicators,  based  Opon  nature  o(  your  program. which  are  applicable  to  components. 
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1.3     PROJECT  6R  SERVICE  DELIVERER-LEVEL  ASSESSMENT  FORMAT 

USE:     To  analyze  and  compare  the  performance t of  the  individual  projects 
and  service  deliverers  (i.e.  /  agencies  and  businesses) ^funded 
through  the  PIC  - 


COMPONENT:  - 


•  Classroom  Training! 


SERVICES  DELIVERS 

ENROLLED 

-* 

.  TUNG.  EXPENDITURES' 

POS.  TERMINATIONS 

ENTERED  EMP. 

Plan 

Act. 

Var. 

Plan 

Var. 

Plan 

Act. 

|Var. 

Plan 

Act. 

-  Var. 

Project  A  . 

V 

% 

Project  B  N 

Project  C  /I 

 j — 

1 

•  -  Deliverer  1 

"  v 

* 

-  Deliverer  2  * 

* 

( 

y 

1  r— 

*  ( 

<r 

* 

 i 

*— 

• 

P 

< 

I 

K 
00 


Artother^ndl.cator^frequently  u^ed  1n  this  area  is  pre-CETA  earnings  vs.  post-CETA  earnings: 


515 
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'     1.4     PROGRAM  ASSESSMENT  QUESTIONS  , 

.  USE; 

To'  facilitate  PIC  prbgrammatic  review  and  replanning. 

s 

% 

'     J  QUESTIONS 

Were  your  i'nitial  planning  assumptions  valid? 

\ 

Were  youl^pitial  goals  and  objectives  realistic? 

*  9 

• 

3. 

♦ 

Was  your  overall  programmatic  thruSt,  or  mix  of  services 
and  projects  sound? 

• 

*  — . 

A 

A          4- U>  a >* o    anw    pnmnnnon  4-  c    r\"F    unnr    nrnrrr^Tn   whl  rh    should    be  » 
Ale    ulicic    cLIiy    Uvjni^uiicii  uo    \jl    yuui    ^ji  u^i  cuu   wiixlh  onwuiu 

dropped?  /  Given  increased  support? 
*  * 

■       X        ■  : 

5. 

Are  there  any  service  deliverers  who  are  ineffective? 
Highly  gffectiv^? 

*~ 

i 
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2.1    ASSESSMENT  QUESTIONS 


USE :     To  analyze  PIC  organizational  and  operational 
performance. 


QUESTIONS 


PIC  ORGANIZATION 

1.     Are  PIC  goals  clear,  understood,  and  shared  by  members? 

4 


Tl     xAre  ttfe  PIC  membersyelear  on  the  role  they  want  the 
Council  to  play  with  reference  to  the  Private  Sector 
Initiative  Program? 


~3y       Are  the  divisions  of  responsibilities  among  and  between 
PIC  staff  and  members  clear? 


Doestfte  PIC  have  a  complete  statement  of  mission  and 
function? 


4. 


^5.       Is  the  Council  structured  most  effectively  and  efficiently 
to  facilitate  both  decisionmaking  and  program  oversight? 

/ 


9 
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PIC  OPERATIONS  • 

6.      Is. the  PIC  well-staffed?    Are  the  PIC  and  the  staff 
assigned  to  or  hired  by  it  working  productively 
together?  -  . 


~7~.       Is  there  an  active  corfr  of  the  PIC  membership  large 

.  enough  to  sustaih  continuous  4ecisionmaking  and  program 
support  activity?    Can  vacancies  be  filled  in  a  timely 
fashion  and  are  they  filled  using  criteria  that  help 
sustain  and  build  this  core  of  active  members? 

:       '     .     .  .  '   •  \ 


8.      Are  PIC  relations  with  the  Prime  Sponsor  good  enough  to 
allow  the  PIC  to  have  any  leverage  on  the  whole  CETA 
system? 


9.       Is  the  PIC  visible  and  positively  received  in  the  local 
business  'community?  ^  * 


PIC  PROCEDURES  AND  METHODS 

10.     '  Does  the  PIC  have  administrative^ procedures  and  operating 
guidelines  in  place?    Are  they  useful  and  current?  * 

+ 


,  518 

O  TV- 31/ 

ERIC       .  * 


Is  the  planning  process  for  Title  VII  prdferly  and 
does  it  allow  enough  time  to  make  reasoned-  decisions? 


Is  the  planning  process  structured ^ to  include 
individuals  and  institutions  whose ' dfcppo^t  for  the 
final  product  is  important? 


Does  the  PIC  have  a  systematic  method  for  monitoring 
program  performance  and  taking  corrective"  action  as 
nebessary?  "  f 


Does  the  PIC  have  a  systematic  plan  for  evaluating 
the  effectiveness  of  each  activity  and  its  total  • 
prpgram?  '  -  4 


Is  the  evaluation  data  which  is  collected  sufficient 
in  quantity  and  quality  for  PIC  decisionmaking? 


IV- 3  2 


2.2     DEVELOPMENTAL  NEED  QUESTIONS 


USE:     TO  identify  priority-  areas  in  which  PIC  needs  to 

develop  its  organizational  an*  operational ' Capabilities . * 


QUESTIONS 


1.  Is  there  £  need  to 'reorganize  or  to  restructure  the, 
•  '   . Council?  • 


"2"^    I?  there  a  need  to  clarify  staff  and/or  merpber  i;oles 
and  responsibilities?  - 


>.Is  there  a  ftjeed  to  improve  staff  or  Council  performance 
either  in  terms  of  work'^output,  relations  with  o,ne  another, 


or  extternal  rd^ations^^-^ 


Is  'there  a  need  to  develop  new  or  more  dejbailed  administra- 
tive procedures  and,*  operating  guidelines?   .Jp  r 


5.   •    Is*  there  a  need  to  upgrade  the  PIC  planning,  Management, 
and  ^valuation,  process?       *,  •  * 


#  • 


ERIC 
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3.1     EGS  NEEDS  QUESTIONS 


'*  USE  i    To -identify^  priority  EGS  service  needs -within  the 
community. 


QUESTIONS 


Is  .there  a  lack  of  awareness  of  PSI£>? 


2.  .Is  there- a  negative  impression  toward  governmental 
*        employment  and  training  programs? 


ERLC 


Do  good  job  opportunities  exist  within  the  local  environment 
which  go  unfilled  because  of  lack  of  contact  between 
governmentally-sponsored  programs  and  local  business  i 
establishments?  • 


Is  there  a  need  to  help  create  43obs  or  to  develop  the 
economic  climate  of  ttie  community? 


5.   ■    Is  there  effective  cooperation  and  communication  among  thase 
.  agencies  which  constitute  your  local  employment  aqd  training 
network? 


iv-a4  521  . 
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■ft  4 


.Hav^  governmental ly-sp<3nsored  programs  been  designed  to 
be  d^pectly  responsive  to  the  needs  of  local  businesses? 


7.4     What  do  local  employers  feel  inhibits  their  employment 
of  the  structurally  unemployed? 


What  services  can  be  provided  to  enhance  the  local 
business*  community^  receptivity  toward  your  programs? 


ERIC 
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3-2     TARGET  GROUP  SELECTION  QUESTIONS 


USE.;     To  identify  target  groups  to  be  served  through  the 
PIC's  program. 


QUESTIONS 


1.       What  is  the  relative  oyerall  severity  of  need  Of  each 
participant- group? 


2. "Wh^t        thQ  "employability"  of  each  participant  group 
based,  lipqn  education  -and  length  of  employment? 


3  -  ~~t  ,  .What '-j^rceitt age  of  the  participant' groups  is  being 
•r-t''S^^*d'3y  other  employment  and,  training  programs? 


\ 


4.f      Dd, subgroups  htf§/e  particular-  and  special  needs? 

(e^gA,  displaced  hpmemakersj  ex-of fenders;  alcoholics; 
'Vietaafn  ^ets  J 


9 


J  * 


9 

ERIC 
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•3 . 3     TRAINING  PROGRAM  OCCUPATIONS  SELECTION  QUESTIONS 

USE ;     To  identify  occupations  for  which  to  develop  or  sponsor 
training  programs.  .  m 


QUESTIONS 


1. 

t 

What, 
year! 

are  the  total  number  of  projected  openings  for  the 

>* 

- 

.  2. 

What 

is  the  percentage  of  "entry-level"  openings? 

3. 

What 

is  the  projected  growth  rate  for  5-yea^per iod? 

4. 

What 

the  hourly  starting  salaries? 

5." 

What 

are  the  promotional  potentials? 

# 

6. 

* 

What 

is  "the  nature  of  working  environment  and  conditions? 

.  '** 

7. 

What 

are  the  skills  or  training  requirements? 

IV-37'       '  \ 
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USE: 


/ 

3. 4 "'TARGET  GROUP  SERVICE  NEEDS  FORMAT 

To  identify  basic  assist^ice  an/J  service  needs 
of  target  group  members. 


Target  Group 

Major  Barrrers 

Service  Needs 

• 

« 
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4 . 1"  ^MISSION  AND  FUNCTION  ANALYSIS  FORM 

USE:    To  review  PIC's  purpose(s),  role(s)  and 
.  responsibilities . 


ROLE  FUNCTIONS 

SELECTED  KOIiES/ 
^FUNCTIONS    (  ✓  )  • 

Advisor  -  reviewer  and  consultant  to 
.i   local  CETA  Prime  Sponsor 
in  program  planning,  imple- 
mentation, and  review  process. 

•  Identify  factors 'whitfh  contribute 
to  program  effectiveness 

•  Examine  the  performance  of  CETA 
and  the  Employment  Service  * 

%    •    Review  and  comment  on  plans  of  * 
economic  development  agencies. 

• 

« 

—  * 

Broker  -  facilitator  and  provider  of 
'  technical  assistance  for 
connections  between  Prime 
Sponsor  and  employment 
community* 

*  • 

• 

Between  individual  employers  and 
publicly-funded  programs: 

•  Disseminate  program  information  to 
employers* 

0    •    Disseminate  information  on  the 
Targeted  Jobs  Tax  Credit. 

\ 

•  Develop  and  market  model  contracts. 

*  rv 

•  Provide  assistance  to  employers  to 
reduce  the  administrative  burden 
of  programs.- 

^     •    Encourage  employers  to  develop, 
and  to  coordinate  wittT  tr&inipg 
agencies,  job  skill  requirement 
forecasts. 

* 

/ 

•  4 

% 

« 

PIC  ROLE  OPTIONS  DEFINED 

KULK  r-UNUl  -LUNb  X 

FUNCTIONS    (  %S  ) 

Between  employers  (as  a  group)  and  other 
groups  or  rf^titutions  in  the  community: 

ft 

• 

i 

•    Coordinate  Title  VII  programs  with  * 
other  employment  and  training 
activities. 

Develop  linkages  between  school 
and  work. 

• 

•    Develop  linkage's  between  employment 
and  training  and  economic 
development  programs. 

Operator  -  designer  and  deliverer  of 
direct  services  either  to 
participants,  employers',  or 
other  agencies. 

  ,k>  

Programs  for  workeiw\ 

•  Provide  wtf^Tc^of  work  training. 

•  Provide  vocational  skills  training. 
«    Provide  entrepreneurship  training. 

• 

Programs  for  employers: 

• 

*  • 

•    Provide  services  linked  to  CETA 
training  ancL placement. 

• 

•    Provide  economic  development-related 
services. 

#  . 
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PIC  ROLE  OPTIONS  DEFINED  " 

ROLE  FUNCTIONS 

n 

FUNCTIONS  (✓) 

•A 

Programs  for  education/employment  and 
training  staff:  " 

• 

* 

•    Promote  exposure  to  private  industry. 

> 

•    Provide  assistance  in  accomplishing 
specific  tasks* 

• 

H 
< 
1 

to 

•• 

* 

* 

*  * 

• 

• 

1 
1 

• 

* 

• 

• 

> 

9  \m 

4 

• 

I 

• 

• 

« 

ER?C     #  532       »   ;  #  533  -  ,  f 


t 


v 


.5.1     STRUCTURAL  ANALYSIS  CHECKLIST 

USE:     To  assess  PIC's  basic  prganizatiojial  and  operational 
*  structure.  1   ^    v ,  j 


v 


u> 


V 


:           ..     .   -  y 

Doing 
Ofcay 
± 

0 

Needs  s6me 
Improvement 

-   *v<  *  3 

4  ' 

f Needs  mucn  ■ 
Improvement  1 

^Relationships  with  other  CEf A  programs  "are 
positive,, 

*     >  * 

4 

«  *  '  ■  

.Physical  snace  is  adequate  for  meetings. 

i 

# 

*  *  * 

\     '•  -  ': 

Chairpersons  are  sufficient  to  mana^^PIC^ 
efforts*  '* 

\  f  

• 

Council  member  workload^  $re  realistic.  - 

Cpuncil  compos#itidn  is*  repr^sentartiv^of  area 
in<Ju3try m*              •  . 

i 

 —   ;  — —                                             —                      ^  1 — 

The  Council  Metabersbijtf'selection  process  is 
appropriate  to  PlC'needs.  * 

i 

At  least*  51 'percent' of  the  PIC's  membership 
»eomes  from  business.   ,                     «  0      ~  5  , 

The  committee,  and  subcommittee  structure.  -  <$t 
effectively  supports  the  PIC1  s  activities. 

I* 

r  * 

« 

Committee  and  subcommittee  membership  is 
easily  and  f airl^netermihed* 

.  t              -  A  :  

• 

b 

*EXA!*rNING  PIC  STAFF  * 

Staffing  is  adequate  to  perform  PIC  function^. 

J   ■    '     %  ~— — 

#  - 

« 

** 

Division  of  labor  among. staff  persons  is 
jeqtiita^le.     *            ^               :   -        .  v 

r 

** 

•           *  • 

ERIC 
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4 


< 
1 


* 

Doing 
Okay 
1 

-  2 

Needs  some" 
Improvement 
3 

4- 

Needs  raucR 
Improvements 
5 

Adequate  processes  for'  supervising  and  direct- 
ing staff  are  in  place. 

* 

 i-1  

v 

Adequate  mechanisms  exist  for  hiring  and 
replacing  staff. 

• 

L 

PIC  staff  are  trained  to, do  their  jobs  as 
heeded • 

• 

EXAMINING  OPERATIONAL  PROCEDURE 
 —  "  

/ 

*  * 

By-laws  provide'  ad^juate  do.cuAeatatiQn  of  the 
PIC's  operational-procedures  and  metfiods. 

♦ 

> 

Council  meetings  are  scheduled  appropriately. 

• 

Council  meetings  are  run  in  a  way  that  ensures 
qQpcimum  participation  and  productivity. 

t 

> 

 i — " 

The  PIC's  approach  to  selecting  officers  is 
6pen,,fair  and  representative. 

*             t  „ 

 1  

Staff  are  cl^ar  on  their  roles  and  respon- 
sibilities. *  . 

* 

— U, — 

Council  members  are  clear  911  their  roles  and 
responsibilities. 

f  t 

There  is  an  up-t*o-date  piC  organization 
c^art.  •- 

A 

All  Staff  members  have  current  job  descrip- 
tions.     -  -  < 

•I 

ERIC 
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• 

» 

•  :  ^ 

/ 

r  • 

■ 

• 

Doing 
Okay 
1 

'  2 

Needs  some 
Improvement 
3 

4 

Needs  much 
Improvement 
5 

« 

Committees1  roles  and  responsibilities  are 
clearly  documented* 

- 

*  • 
• 

Committee  assignments  are  listed  and 
disseminated,  ' 

• 

• 

fic  start  and  membership  are  clear  on  the 
PIC's  relationship  to  the, Prime  Sponsor, 
business,  community  groups,  etc. 

• 

The  PIC  has  put  an  adequate  system  in  place 
for  planning,  managing  and  evaluating  its 
efforts. 

M 
< 
1 

i 

# 

ERJC  - 

ton  f 

EXAMINING  ORGANIZATIONAL  STATUS 

'i' 

> 

 v  ■  ; 

The  PIC's  statiS  is  consistent  with  overall 
CETA  programmir g. 

• 

The  PIC  iff  able  to  represent  and  foster 
private  sector  irivolvemgnt . 

The 'business  community  has- a  positive  regard 
for  the  PIC.    ^  * 

- 

• 

• 

■  :  1  r 

*       i                                               '*  * 

t.  0 

*  • 
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6.1  ,-GOAL  SETTING  FOlfo 
USE:    To  develop  strategic  goals  for  the  PIC. 


AREA 


i  NEEDS/BARRIERS 

u  


RELATED  GOALS 


EMPLOYMENT 
GENERATING  ' 
SERVICES 


i 


< 
I 


PARTICIPANT 
SERVICES  - 

TRAINING" 
AND  * 
DEVELOPMENT 


PIC 

ORGANIZATION 


ERLC 
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•C.  *  ESTABLISH  PROGRAM  SCOPE 


PME  AIDS  S 


Step  - 


3. 


Define  Appropriate  Tactics 
1;1  Tactic  Selection  Form 
Draft  PIC  Program  Objectives 

2.1  Participant  Training  and 
Development  Objectives  Form 

2.2  EGS  Services  Objectives  Form 

Specify  Project^  and  Scope    %  ^ 

3.1  Participant  Project  Scope  Planning 
Worksheet 

3.2  Cost  Estimation  Worksheets 
— Allowances 

— WageS  and  fringes 


Section  III 
Reference  Pages 


III-143-147 


III-151-153 
III-L53-154 


III-161-165 
111-167^171 


) 


ERIC 
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Hr.  1     TACTIC  SELECTION  FORM 

#  #  * 

USE;    To  determine  and  record  what  tactic (s) 

to  employ  to  accomplish  strategic  goals, 


< 


GOAL 


BARRIER 


PARTICIPANT-RELATED  TACTICS; 


Classroom  Training-Vocational 
Classroom  Training-Employability 
OJT  •  ' 

Upgrading 
Retraining 

Emplqyment  and  Training  Services 
Supportive  Services 
Customized  Training 

Pre-EmplQyment  &  Job  Search  Training 


TACTIC 


EMPLOYMENT-  GENERATING  TACTICS: 


Labor  Market  Analysis 

Marketing  * 
i  %  Economic  and  Community  Development* 

Incenfcd\je  Programs 
,     Linkage  Creation 


ERLC 
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2.1    PARTICIPANT  TRAINING  AND  DEVELOPMENT  OBJECTIVES  FORM 
/  USE:    To  set  objectives  for  participant-related  projects. 


DEVELOPING  PIC  PARTICIPANT  OBJECTIVES 


PARTICIPANT  GOAL:  » 


WHO: 


HOW  MANY: 


WHAT  MEANS  :- 


PRIMARY  RESULT:   .  .  

HOW  WELL:  {e.g.,  average  cost/participant  average  wage) 


/  J 

545 
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2.2     EGS  SERVICES  OBJECTIVES  FORM 
USE:    To  set  objectives  for  EGS  projects, 


DEVELOPING.. PIC  EGS*  OBJECTIVES 


EGS  GOAL: 


TIME: 


r 


QUANTITY : 


QUALITY: 


COST-; 


54G 


9 
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t 


% 

t  a* 

1 

• USE^:  To 

-                  -  • 
• 

plan  scope  of  participantg.sdrvin< 

j-  projects. 

*  * 

V 

PR*OJECT*SCOPE^PLANNI^G  WORKSHEET 

*  * 

,  TARGET  £ROUP' 

, DESCRIPTION 
OF  MAJOR  ' 

PROJECT 

PRIMARY  „ 

OTHER  SERVICES  NEEDED-  T?0  ACHtfeVf 

ESTIMATED 

Description 

No. 

Carrier  (S): 

OBJECTIVE 

PROJECT? 

OBJECTIVES  (enter  names) 

COST 

• 

* 

■ 

a 

w 

0 

,   .  ... 

- 

* 

# 

• 

• 

• 

• 

* 

%  1 

* 

V 

1 

U 

# 

A 

* 

4 

\ 

* 

0 

• 

* 

*  * 

**  t 

\ 

•  4 

*  < 
I 

en 


er|c 
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3.2     COST  ESTIMATION  WORKSHEETS, 

USE:    To  project  alliances  \  and  wages  and 

1    fringe  costs  for  participant  serving., 

projects.,  *  v  fc  j  * 

« 


Format  for  Calculating  Allowances 

*    *  * 

Pro  j  ect/Gooiponent 
Basic  Training 

r 

• 

No.  Participants 
Qualifying 

% 

Average/ Basic 
.  Allowance 

Per  Wfek 
$  X 

NO.  of 
Weeks 

* 

$ 

Allowances  for^ 
Dependents 

 V   . 

No.  Participants 

Qualifying 

X 

Number  ijteeks      Amt.per    Avg.  No. 
of  Trailing  •      Depend,    of.  Depend. 

X  $           X  - 

•  > 

$  - 

 .  '  r- 

Meals,.  Trans- 
portation, etc  . 

No.'  Participants 
•  Qualifying 

X 

No.  Weeks 
Parti cip. 
>in  Trng., 

X 

Avg. Allow, 
for  this 
Purpose 
per  Week 
$ 

^ — 

$ 

TOTAL  ALLOWANCES  ■ 

$ 

IV-S3 
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Format  for  Calculating  Wages  and  Fringes 
Project/Compbnent  f 

(1) 

Occupation 

-  (2) 

NuuiDer  or 
" Positions 

(3)  (3) 
Avg.  Hours  of 
.^Employment  or 
Training 

(4) 
Average 
Hourly 
Wage 

(5) 

Total  Wages  for  . 

Occupation 
Cols .  2  x  3  x  4  i 

(6) 

Fringe 
Benefits 

•  • 

i 

* 

 ^     ■    .r  ■  

Total  Wages 
$ 

Total 
Fringes • 

$ 

~  "  

ERIC 


IV- 54 


'  t 


i 

\  *  •  \ 


D.   DEVELOP  PROGRAM  OPERATIONS  PLAN 


Step 
1. 


PME  AIDfe 


Structure  Program  Delivery  System 

1.1  Project  Design  Formats 
— Project  Planning 

— Activity/Service  Planning  Format 
— Management  Plan  Format 

1.2  Project  Delivery  Questions" 

1.3  EGS  Questions  and  Control  Format 

Establish-  Performance  Standard^ 

2.1    Standards  Setting  Form 

Secure  Necessary  Assistance 

3.1    RFP  Contents  Checklist 
,3.2    Proposal  Review  Criteria 


•   Section  III 
Reference  Pages 


III-177-179 


III-179-182 
111-182-184 


III-187-193 


*  III-198-201 
1-202-204 


3.3    Current  Contractor  Review  Guidelines 


and  Fortn 


4.     Finalize  and  Approve  Operations  Plan 

4.1  Program  Operations  Plan  Outline 

4.2  •  Final  Cost  Plan  Checklist 

4.3  Sample  Work  Statement  Format 


IJI-204-205 


III-211-212 
III-212-213 
III-211-214 


1. 1     PROJECT  DESIGN  FORMATS 

USE:     To  plan  and  develop  designs  for  projects  £6^- 
be  sponsored  by  through  the  PIC* 


1.     Project  Planning  Format 


Project: 

* 

MAJOR  ACTIVITIES/SERVICES 

i                    PERFORMANCE  STANDARDS 

c 

* 

 ■  &z  

2.     Activity/Service  Planning  Format- 


PROJECT: 

MAJOR  ACTIVTTY/SERVIC1 

PERFORMANCE  STANDARDS:       ^  -j 

1+    ACTION  STEPS 

2.  RESOUtiCES/ASSXS. 
REQUIRED 

3.  TIME 
FRAME 

4.  PERSONDAYS 
ALLOCATED 

-  f 

»  * 

\ 

m 

3.    Management  Plan  Format 


1 

* 

ACTIVJTY/TASKS  f 

STAFF  RESPONSXBI£  - 

\  OF  TIME  TO  ACTIVITY 

• 

k 

• 

la 

« 

* 

• 

ERIC  • 

» 

c 

« 

■  t 

» 

.     '  '  ?  • 
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* 

m 

% 

# 

• 

1.2 

PROJECT  DELIVERY  QUESTIONS^ 

USE;     To  determine 
coordination 

basic  projejct  'delivery  arrangement  and 
mechanism  for  all  PIC  projects. 

* 

QUESTIONS 

A.       Participant  Services: 

1.    "  How  will  your  PIC  handle 
— outreach? 
— recruitment? 
*      — intake? 

--assessment? 


~2~.       What ,  if  any,  provisions  will  be  made  for 
*  — Participant  special  development  needs? 
--Participant  support  service  needs?  . 


Will  your  PIC  undertake 

— placement? 

— job  development? 

--program  termination  activities? 


4.       Will  your,  PIC  give  follow-up  assistance  to 
*  program  services? 


IV-58  <J2.J 


-jrv^L^GS  QUESTIONS  AND  CONTROL  FORMAT 


USE:    To  establish  methods  for  overseeing  and  controlling 
EGS  projects, 

QUESTIONS 


1.      What  end  products  or  results  will  be  ^foduced?     In  what 
time  period  or  by  what  date? 


1  2* 

What 

 E-  — 

interim  products  will  be 

* 

9 

produced?    By  when? 

1  — ^ 

1  3" 

What 

 a— — ■  

major  activities  will  be 

♦ 

accomplished?    By  'when? 

1  4* 

What 

 ■  7  

are  project  benchmarks? 

When  will  they  occur? 

■ 

■ 
1 

* 

— *  1 

«What  are  the  major  tasjcs  or  steps  necessary  to  complete 
each  project?    What  is  their  timeline? 


ERIC 


PROJECT/ACTIVITIES 

Activity  1 
ff  Activity  2 


1      2    -3    _4      5      6  17 


MONTH 

ft 


40 


11 


12 


A  3  major  benchmark 

^  =  deliverable  product 


IV- 60 
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2.1     STANDARDS  SETTING  FORM 

1 

USE:  To  develop  performance  Standards  for  PIC 
— ~      programs,  components,  and  prefects. 


Assessment • Level  (Program/Component/Project) 


Assessment  Area  (Pro jectTActivity/Services) t 


INDICATORS  • 
(What  to  Measure) 

STANDARDS 
(Level  of  Perf.  Expected) 

» 

* 

« 

0 

.IV-61 
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3.1     RFP  CONTENTS  CHECKLIST 

US&:    To  structure  effective  Requests  for  Propo 
to  assist  in^the  delivery  of  PIC  projects 


Q  Project  Title?  '  t 

O  Describes  project 

O  Brief  description  of  service  to  be  provided 

□  Indication  of  competitive  bidding 

□  Closing  date  for  bids f  /      r  • 

P]  Program  Information 

□  PSIP-CETA  legislation 

□  PIC  in  general  * 

□  Your  PIC  conberns/per^pMectives  and  mission 

□  Relevant  goals  from  your  strategic  plan 

PI  Project  Objectives 

m    □  Services  to  be  provided 
Q  Purpose  of  project 

□  Specif i.c  project  objectives 

'      □  Obstacles/barriers  at  vtfiich  project^is  aimed 

□  Assumptions  underlying  service  approaches 

I"!  Scope  of  Work 

□  Sub-objectives  of  project 

□  Activities  to  be  performed 

□  Services  to  be  provided 

□  Characteristics  of  target  groups  * 

Q  Performance  Measures  and  Evaluation  Criteria 

□  Measures  for  project  monitoring 

□  Measures  for  evaluating  performance 

□  Evaluation  criteria  for  each  project  component 

□  Performance  standards 


j  |  Performance  Schedule  *  \ 

□  Major  activities 

□  Achievement  le\ 

Q  Specific  dates 

*  < 

Management  and  Reporting  Requirements 


□  Achievement  levels  J' 


□  Required  reports 

□  Required  administrative  procedures 


557 
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.fH  PIC  Policy., and  Regulations 


□  Contracting  procedures 

□  Type  of  contract  to  be  awarded 

□  Method  of  payment 


\ 


\  ■ 


r 


ERIC 


7   3-2     PROPOSAL  REViEW  CRITERIA 
.  USE:    To  analyze  r^spigtises  to  Requests  for  Proposal 

Following  is  a  list  of  in-depth  proposal  review  criteria 

Program  Impact/Outcome  -  Potential  contribution 
-  of  the  program  toward  the  resolution  of  the 
problems  experienced  by  the  eligible  populatidn, 
and  ^he  program* ^  potential  contribution  toward 
the  attainment  bf,PIC  goals. 

1.    Will  the  proposed  program  result  in 
•    placements  in  the  private  sector  at 
a  rate  acceptable  to  the  PIC? 

2,.    Will  the  proposed  program  result  in 
proportional  placement  in  jobs  for  all 
significant  segments? 

*  •>  ' 

3.     Will  the  proposed  program* facilitate 

and  restflt  in=  retention  of  participants" 
by  employers  after  placement? 

4'     Will  the  proposed  training  and/or^ser- 
vices  enhahpe  the  upward  mobility  and 
promotional  opportunities  of  partiCi- 
*  pants? 


1 


Program  Design  -  Potential  success  of  the 
£rog*am  design  in  the  attainment  of  the 
described*  outcomes/impact. 

1.  Dotes  the  .proposed  program  s&tisfy  one 
or  more  of  tha^riteria  for  projects 
to  be  funded  urSTer  Title  VII  within 
th%  PIC?  '  x 

2.  Does  the  proposal  c&ntain  a  feasible 
design  and  rationale  describing  who 
will  do  what,  when,  where  and  how 
(i.e.,  program  activities  and  partici-. 

.  pant  flow)  ?  •' 

*  « 

3.  If  the  cooperation  of-  other  ,organiza~' 
<  tions  is  nec.essary  %to  successfully 
implement  the  program,  does  the  pro- 
'posal  describe  the*  coordination  with 

# those  other  organizations? 


55<) 

IV-64 


4.  Ddes'the  proposal  describe  adequate 
and  potentially  successful  procedures 
for  outreach,  recruitment  and  assess- 
ment of  participants  that  will  ensure 
enrollment  of  significant  segments  at 
the  proposed  rates? 

5.  Does  the  proposal  adequately  describe 
training  for  occupations  which  are  in  ' 
demand'  and  explain  how  persons  trained  ' 
will  be  successfully  placed  into  such  ;\ 

jobs?  ,  : 

6.  Does  the  proposal  adequately  describe 
how  t;he  offeror  will  provide  the  *  support' 
necessary  to  sustain  participants  through- 

'  out  the  straining  in  order  to  reduce 
attrition? 

Program  "Cost  -  {low  do  the*  following  cost 
factors  compare  with  competing  proposals* 

1.  Cost  per  participant  enrolled  (total 
program  cost  minus  income  transferred" 
to  participants^ .  % 

2.  Cost  per  placement  (t^tal  prdgram  cost 
minus  income  transferred  to  participants) 

3.  Does  the  proposal  adequately  explain 
and  justify  the  costs  of  the  pro.gr am? 

Management  Plan  -  Ability  of  the  organization 
to  implement  the  proposed  program  and  attain 
the  stated  objectives. 

1.  Does  the  proposer's  past  relevant  ex- 

•  perience  indicate  a  potential  for  suc- 
cess fui  management  and  implementation 
of  the  proposed  program? 

2.  Does. the  proposer  demonstrate  success , 
« or  reflect  a  potential  for*  success  in 

providing  the  described  seryices  and/or 
training?. 

3:  Does  the  proposer  demonstrate  success, 
or  reflect  a  * potential  for  success,  in 
serving  the^  eligible  population? 

4.  Does' the  management  plan  adequately  and 

*    feasibly  describe  the  necessary  activities 
events  and  timeline  for  successful  imple- 
mentation of  the  program? 

560 


3.3     CURRENT*  CONTRACTOR  REVIEW  GUlDEIiIflgS  AND  EORM  - 

'      .-/.  '•  ■  •  '     '  ' 

USE:.    To  review  and  assess  the  .performance  of  current  PIC 
.    ~      contractors'.  '■  c 

•  i     '•  *  <      *  -  •••  •  M-  '••  - 


• 


.    \   Is  the  training  ^entity r  well-e'stablist\ed?  _  ./ 

f.        Does  the/tra sking  entity  have^adequate-^cilities? 


Will  it  provide  counseling  and  placement  ^  ^  - 
se««*ces?        ,        ^    ±  <(  ^  *    ?  *  S  , 

Has  the  Tnirrieulum  bg^;i  developed  tfith  private 
!       ■  •         ,     sector  irgjt?  •      *V     "  ^  . 


-1 


1  • 


*  Boes  ^the  -  entity  have  adicj^fate  acpountiri^  %pfdcedui:.es?" 

****  4 

What' are-- criteria  for  instructors?  *    *  • 

Will  a^tud^Lr^l  a#d  skin  training' be  provided? 

"Will  training  qualify  CETa' participants'  fo^  job'  ' 
openings?  \  :     *  <  _ 

.What  are  procedures  for  placing- -workers  in  " 
urisubsd i  H  i  zftd-^oMa?  ^  ,  t.  ^ 

•'  V'- 


7^ 


4^ 


3  " 


4 


•     /     IV-66     '  •  s\  j       #  - 


.EBJC  .  ■     ;  A    ■  ■> 


Projected  Outcomes  for  FY'81  ^ 
Description  of  Perf or»#nce-hy  Current  Contractor 


Jurisdiction:  

Program  Activity*   


Planned  t.  ,pr  ^ 
Perf  ^  Stand,  ~ 


Actual  Variance* 


— Total  Ept< 

1.  Minorities' 

2.  Females  -* 


v 


*  1  .V 


3.    Youth  (21  Ja  under)  ~  '  *'  '     ^-   1  -V*  'liL  il 


_  Total  Terming  Itofw; 

Entered  Unsubsid  iz*d  Employment  ^  '        *   *  4_ 


}  V 2.    'Additional  Positive  Terminations 


3.    0therr7emf  nations  ^ 

*  ~  » 


C.    Entered  Uifsubsidfzed  Employttejnt 

•ini  a  Private  Sector  Job    ^       -  .  .     '  -     ,  ?. 


J 


{jSE  i    rTo  structure  cdhtent  fcjr  PIC 1  s<  ProgiJcBfT  Operations 
Plan. 


4.1     PROGRAM  OPERATIONS  PLAN  OUTLINE 


PIC  Program  Performance  /Standards  -  ■ 

PIC  Administrative  Stjruqrture  .and  Organization 
PIC  Operating  Procedures  , 
PIC  Program  Delivery  System 


.    PIC  Projects 
,PIC  Management  Plan  and  Proc^PWes 

PICt'Evaluation  Plan  and  Procedures 
•Budget  Allocation  and  Accounting  Procedures- 

■    \  ■  .  . 


^  •  4.2     FlflAL  COST  PLAlt  CHECKLIST  ' 

USE:    To  review  final  program  costs  to  ensure  that  they  ajre 
—    -within  original  planner  range  and  PIC's  final  budget 
allocation.  . 

PI     Document  all  final  projects  costs  /     *  s 

P  *,  Compare  final  project  costs  to ^original  estimates 

Q     Note  differences  ,*  ^ 

-  _-„.  #  •    ;      - 

("1     Display  all  line  item  costs, 

Check  to  make  sure  that;. funds  are  planned  to  generate . 
•    .  services  at  maximum  le^ls  of  efficiency  % 

f  LJ     Arrange  co§ts  in  proper  categories 

[H    'Check  to  make  suje*»  'tjhat  ^ali  projected  costs  are  in 
accordance  with  f ederal  regulations-: 


□  Participant  related  -  at  least  50% . 

□  E6S  -  up  to'  30%  • 

dt  Administrative  -  no  more  than  20% 

™  ,    »        \        * .  ■ 

TChe<jk  to  be  sure  that  total  costs  don't  exceed  total 
allocations  ^      _    ,  # 


1  -      •  *T« 


]  -'  * 


4.3     SAMPLE  WORK  STATEMENT  FORMAT 
USE;    To  serve  as  a  model  for  project,  work  ^btements, 

YOUTH  TRAINING  U^IT 


Number  to  be  served: 
6^nonth  period: 
—Type  of  ^Training z  » 


60  slots 


November -1981  through  April  1982 

Various  including  Clerical, 
Merchandising ,  Brick-laying ,  # 
Electrical  Repair,  Food  Service 


1-  '  vWhat  service  the  component  pi+ovides 

The  PIC  will  provide  allowances  and  in  some  cases 
tuition  payments  for  persons  participating  in 
occupational  training  at  ^^rious  local  prtVate 
sector  training. "in&titutions .  . 

«  «* 

2 Client  assignment  standards 

v 

Counselors  w\XT  assign  persons  between  the  ages 
of  17-22  who  show  prerequisite  talents,  a  degree 
'  of  dexterity,  and  basic  education  to  successfully 
.  complete  training  within  a  26-week  period' or  less., 

3-       Whafr  wprk  is  jio  be  done  ' 

'  '  \ 

The  Youth  Training  Coordinator  will  \ 

t  * 

-.■a.  jDe^ejLop  training  sites,  as  required. 

b.  yfcoor&inate  the  flow  of  participants  into  and' out, 
Tv    r  of  tjograro.      -  \ 

c.  „    Analyze  -ana  report  oh  the  ef f ectiveness^of  the  program. 

d.  Monition  training  sites  and  applicant  progress.    '  * 
*  e.  Monitor  the ^payment  of  trainees. 

The jfcounseling  staff  will  .  —  s 

I  -  .  .  . 

a.  7  'Select  and  rel§er  eligible  participants. 

b.  /     Maintain  follow-Up  as  required. f 

Ttie.  training+institiiti^ns.will 

•  « 
a.      provide*  graining  in  the  various  skills  areas.- 

Aid  iji  the  placement  of  trainees. 

Provide 'information  on  participant  progress  to  tHfe 
projeq.tr  coordinator • 
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Worjt  Schedule  *  • 

a.  Enrollment  in  training  programs  will  be  accomplished 
on  an  open-ejitry  basis,  according  to  individual 

needs.    S  ^  .  ^ 

b.  Training  will  be'  completed^within  a  maximum  of  26 

we  e  k  s  *■* 

c.  All  participants  will  be  paid  on  a  weekly  basis, 
attendance  records  will  be  processed  weekly. 

d.  Follow-up  contacts  will  be  made  tos  necessary, 
but  not  less  than  Once  per„  month. 

How  the  worJf"ls  to.be  coordinated      %  : 
Support  "given     '  ^  ^ 

.a.      AVouth  training  coordinator  wi,lj  inform  counselors' 
of\vailable  slots. 

b.  TheVoordinator  will  .insure  that  counselor*  are  awar 
•  of  status  changes  and  the  need  for  special  .follow-up 

c.  "She  coordinator  will  assist  counselors  in  obtaining 
follow-up  and  progress  reports  ,  -  - 

d.  "     The  cooyainator-  will  monitdr  payroll,  and-attendance," 

e.  The  coordinator  will  insure  proper  implementation  of 
.drogram.  / 

V* 

Support  received-  •  'I  * 

a.  '  Counseling  staff  will  inform  the  coordinator  when  _ 
'"slots  are  desired.  \.  ,  .  . 

b.  Counseling  staff  will  refer  eligible  applicants. . 

c.  Counseling  staff  wiLl  provide  support  services  for 
trainees  as  needed. 

/ 

•  Administrative  procedures  ,        r  •. 

The  training  agency  will  prepare  and  submit  all  Joxm,  . 
-internal  records,  logs  and  reports  required  by  the  youtR. 
trrfning  coordinator..   The  coordinator,  will  Provide  same- 
to  the  director  of  employment  and  training  .programs  -  ana 
to  counselor- sujfervi sore  when  necessary, 

Performance  Standards  ■ 

The'  following  standards  will  be  used  to  measu 
in  this  component: 

•  V 

a.  All  training  slo.ts  shall.be  dejfe  ^ 
private  or  public  training  agenciasA^  .  'Nr 

b.  Not  more  than  10  percent  of  .participants,  shall 

•  be  terminated  for  reasons  other  than  tr^^ 

tovanother  status.  *  \  ^ 

. ,  oc  r,€  nnmnlptiona  will  be 


c. 


Npt"less  than  85  percent  of  completions  will 
placed  in  jobs  as  a  result  of  the  trainifcg^ 
Sot  less  thanSo  percent  of  the  occupat*»hal 


intng  conduc  eSwIIl  be  completed  in  2  S}  weeks 
less.  •  ,Jr  >*> 
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e.       The  Prime  Sponsor  shall  require  the  accurate, 
legible,  and  timely  submission  of  attendance 
records  on  a  weekly  basis, 
*  f .       The  Prime  Sponsor  shall"/ require  submission  + 
of  complete  and  accurate .trainee  "progress 
reports  on  a  monthly  basis.  % 

Monitoplng  arrangements 

Standard  (a)"  shall  be  insured  by  the  youth 
training  coordinator's  personal  .review  prior  * 
to  commitments  ^ 
U       Standards  ip) ,   (c) ,  and  (d) . shall  be  monitored 

♦by  the  director  of  planning  and  'evaluation  through 
reports  required  for  this  purpose, 
c.      Standards  (e)   anjd  (f)   sh^U.  be  monitored  by  the 
youth  training  coordinator  ancj'  director  of  work 
training  - 

Staffing  ^         .  * 

ining  coordinator  (full  time) 
1  payroH  clerk  (1/3  time)       "         — -  r 
♦1  secretary  (1/3  'time) 


J- 


*f   ,E.   DEVELOP  MANAGEMENT  (MONITORING)  PLAN 


PME  AIDS 


Step 


Section  III 
Reference  Pages 


s 


Establish  Management  Review  and  Control 
Process  x  ■ 

1.0.    Operational  Review  Process  Questions 
1.2    Responsibility  Assignment  Questions 

Determine  Management  Information  and 
Re^Q^tLinq  ^ffiecfs    . 

2.1  Basic  Management  Information  Needs 
Questions 

2.2  Management,  Information  fend  Monitoring 
Planning  Worksheet         J  t 

Design  Management  Reporting  System 

. 

3.1  .Basic  Monitoring vFormat 

3.2  Trend  Analysis  Form 

3. '3    Comparative  Analysis  Form  ^ 
3.4    Trend  and  Comparative  Analyfis  Form 
3. .5    Monthly  Participant  Activity  Report 
Form  *  ^  ' 

Monthly  Services  Assessment  Form 
Sample  T^end  Analyses  Management 
Reporting  Display  Chart 


3.6 
3.7 


III-221-224 
III-224-228 


III-231-Z33 
III-233-239 


III-245-246 
III-246 
III-246 
III-248-250 

111-246-24^ 
111-2^6-24^ 

III-246-248 


— 1  / 


■I 
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1 .  1    JBPF.RATTONAT.  REVIEW  PROCESS  OUESTIQHS- 


USE:  To  determine  basic  procedures  and  method*  for  overseeing,  • 
diYecting,  and  controlling  PIC  operations  and  program  performance. 

QUESTIONS 


By  answering 'the  following  questions,  you  can  lay  the  ground 
work  for  your  PIC's  basic  monitoring  procedures. 


*  What*  information  is  absolutely  essential  for  monitoring 
.  performance  and  managing  the  PIC's  program? 

*  What  are  the  sources  of  this  Information? 

*  How  can' this  information  be, collected? 


*  What  proce4ures  are  necessary  for  analyzing  the 
information? 

*  When,  how  often j,  and  in  what  manner  should- the  information 
t  be  reported?  . 

*  Who  will  review  management  reports? 

;  *  How  will  corrective  actions  tie  implemented?  , 


r 


IV- 7^ 


^^^^^ ^^^^^^JESPONSIBILITY^SSI^ 

USE;   'To  V*jstablish  individual  responsibility  and  accountability 
s    f or  ^Rnplementing  PIC's  operational  review  process. 


■%  QUESTION^ 


.By -answeringL  the  following  questions ,  you  will'  have  the. 
information  necessary  to  determine  basic  management  review 
responsibilities  for  both  PIC  staff  and  members. 


*  Who  will  have  ultimate  responsibility  for  administering 
the  monitoring  plan? 

%  *^Whd  will  supervise  monitoring? 

'  V 

*  Who  will  prepare  monitoring  and  management  reports?  1 

.  %  '         '     *  -  %  \ 

*  Who  will  be  involved  in^the  analysis  of  monitoring 

reports  and  in^  the  interpretation  of  findings? 

*  What  will  -the  Council's  role  be  in  the,  monitoring 
•  process? 


USE:     To  determine  the  general  types  and  quantity  of  data 

effective  program  management, 


QUESTIONS 


eric  ; 


2.2     MANAGEMENT  INFORMATION  AND  MONITORING 
PLANNING  WORKSHEET  • 

<  -  ■  ™  * 

USE:    To  determine  specific  information- needs,  sources, 
method  (s)  and  frequences,  o£  data-  col lectionrf or  "  *' 
each  PIC  project. 


Component/Pro j  ec t : 


0««*CTIV£  % 


NtOWATOM  Of 


MCOUMtMOtO  • 

*  f*LOUtMCT 
% 

*  NO  OPttMAL  JOl/UCt 


:  / 


CUiMNIl  * 


■5 
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USE; 


.    /       T.I     BASIC  MONITORING, J-QRMAT,  ' 

.To  chart  and  review  PIC,  program  'performance , 


\ 


ERLC 


Area 

Performance  Indicators 

?  !  ■ 

Actual 

»  Plan 
 ,  

%  Variance 

■ 

^  -  — 

• 

i 

A 

i 

«-/* 

•  * 

f  ¥ 

1  / 

s 

• 

* 

• 

1 

<* 

V 

a 

• 

n 

i 

%  . 

• 
• 

» 

4 

$ 

«                    *  c 

♦ 

>                              -  * 

•         •*     ♦  . 

 a  -n   

t 

'* 

'    *v  <  •       .    ■  ' 

USE:    To  analyze  performance  *    3.2    TREND  ANALYSIS  FORM  ..  '  * 

over  time.  ••  _  „__  "  

•*  TITLE  VII 

PERFORMANCE  INDICATQR  TRENDS/  SIGNIFICANT  SEGMENTS*  PERCENT^bE  OF  •PROGRAM  ENROLLMENT* 


V  :  : 

i 

SIGNIFICANT  SEGMENT 
ENROLLMENT  CATEGORY 

lst-Quartet 

2hd*Quarter 

3rd  Quarter 

■ 

4th  Quarter 
FY  80 

* 

Pl.% 

** 

Act.% 

%  of\ 

Dev. 

* 

Pl.% 

■  ** 
Act.% 

%  Of 

Dev. 

i  * 
1  PJ,.% 

*  * 

Ac  t .  % 

%  of 

Dev . 

Pl.%* 

Act.% 

*   OP  . 

Dev . 
T 

•  r 

Male 

> 

• 

i 

Femalfe  -                     1  , 
• 

- 

-  19  &  Under    *  , 

 * — 

- 

i 

20-21-  '«■" 

'  » 

• 

j  * 
i 

1 

'  22-44 

  -  "               '                *■   ■  / 

* 

— I — 

i 

i 

Vs,                J     .!;<:  ■ 

t    |   .  W  ^  1- 

4 — 

*  • 

*  1 

l 
i 

! 

55  fc  Over       ,               '       \  * 

f 

j 

i 

1 

! 
• 

1  • 

r— f- 

—       *  *  * 

w.  -  (n.h:)  » 

j     .  % 
j 

1 
i 

/  -  * 

v  B.   -   (N.H.)             *                §  ¥ 

i 

i 

Hispanic 

1  

r 

• 

i 

i  ■ —  '  -  . 

A.I.   t  AN. 

!  

» 

A.   &  P.I. 

— r —  »  1 

1 

* 

H 
< 
I 

00 

o 


Plan,  percentage:  Use  demographic  tables -in  Ahnual  Plan  or  compute  from  PPS  (Part  1VA-U  7  (Part  IA),;  1 
••Actual  percentage:    Compute  from  Q5£C  (sig.  seg. ) 7 (Total  pa/tic  enrolled) 


er|c 
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3.3    COMPARATIVE^ANALYSIS  FORM  ■ 
USE:t  To  compare  and  assess  performance  among  PIC  program  components . 


COMPONENT:  . 


Clagsrodfn  Trajning 


Quarter 


SERVICES  DELIVERERS  / 

%  

ENROLLED 

TRNG.  EXPENDITURES  i 

POS.  TERMINATIONS 

ENTERED  EMP.' 

",PW 

Act. 

Var. 

Plan 

Act.. 

Plan 

Var. 

Plan 

Act. 

Var. 

Project  A  ,  'A\- 

Project  B       J    |^   "  .  ' 

4 

* 

• 

'Project  <?'  '-l^^P- 

V 

j 

9  J 


-  Deliverer  1  jr.    .  * 

• 

• 

 %  i    \  :  

-  Delivered  2 

 - 

.  / 

 h 

....  '  -  .     \- .  . 

* 

•    '  j 

Another  Indicator  frequently  used  In  this  area  1s.j>re-CETA  earnings  vs.  post-CETA  earnings. 
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-  3.4*    TREND  AND  COMPARATIVE  ANALYSIS  FORM 

*  w"  

USE:    To  analyze  and-  compare  performance  over  time  among  PIC 

"     program  components. 

*   '   PERFORMANCE  INDICATOR  TRENDS:  EXPENDITURES* 
.   •  *  TITLE  VII 


'J 


1  1 

PROGRAM  COMPONENT  *            <  | 
*                                          1  t 

i 

4th 

Quarter 

1st 

Quarter 

2nd       j  3rd 
Quarter    j  Quarter 
! 

4th  j 
Quarter 

\ 

CLASSROOM  TRAINING 

% 

% 

% 

'% 

1 

—   r— r  

> 

OJT 

• 

< 

\ 

*  » 

SERVICES  TO  PARTICIPANTS 

■  4  1 

:  - 

*  < 

EMPLOYMENT  GEN/  SERVICES 

0  . 

1 

'  OTHER 

J  * 

OTHER  V 

t 

TOTAL  EXPENDITURES 

< 

4 

1 

Upgrading 

Retraining         ,  ^/ 

• 

* 

i        .  , 

*•  Enter  percent  of  program  funds  expended,  compared  to  planned  budqet. 
R^f:    Financial  Status  Report. 


573 


573 


USE: 


3.5     MONTHLY  PARTICIPANT  ACTIVITY  REPORT  FORM 
'  '  '. 

To  present  data  regarding  participant  progress,  in  PIC  program. 


< 
I 

00 

u> 


8 


M 

-I 

'J 

O 


,  O 


2 


da 

r 


REPORT  #1 
Report  for  Period  Ending 


MONTHLY '  P10C1AH/ACTIVITT.  EEPOgT 


l~f   TOTAL  OTA  PI0G1AK 


/~7  ACTIVITY:  ^_ 


ASSESSMENT  INDICATORS 
• 


1.    Totirl  Cojtg  (2  +  3) 


2.    UguUr  Cottt 


II    AdalnUtratlvt/Allocifd  Co«tt 


4.    Tof  1  CfoUtlva  Participant*  Ui  ♦  4b) 


On  iQTd  Uilnnlni  of  Grant 


■•v  Participant* 


Tbtal  Tarmlnatlc  

a.  Tnf  1  Placa— nta  (Sad)  ♦  Sa(2)) 


b.  ♦  c) 


♦  tndifct 


b.  Total  goaltlva  Earlv  Tar«in«aa 


c.  Qbbar  Taralnatlon* 


6.    Current  Enrol  1—nt  (4  -  S) 


7.  *+Niafear  Collet  lot  Activity 

8.  »M«.  Collating  Job  Entry  Period 


1(T. 


11. 


12. 


13.  Poaltlva  CogeUtljo  (7  +  Sb  ♦  S  k  100)** 


14.  Plac 


Utt  (5 


TooT 


15.  Dlract  Flictn nt  hti  (Sa(l)  ♦  3»  »  100)* 


16.  Job  Entry  ComplttloQ  lit 


att  (8  ♦  5«  k  100)* 
tlo  (3  I  1  x  100) 


18.  Amm  Coat  ptr  Participant  Q.  ♦  4) 


19^  Avtrlgt  Cost  par  Placa—nt  (I  ♦  5a) 
20.  Avir.  Coat  par  Poaltlva  Coaylatlon  (1  V*_+_ 


PLAN 


JDDL 


ACTUAL 


ptr  Potl 
Tl~Avarata  Coat  ptr  Kan-yaar 


Z  VAIIANCE 
FI0H  PLAN 


XXX 


-Ba- 
rn 

xxx 


XXX 


XXX 


JSL 


pnponuNCE 

STANDARD 


xrr 


WAENINC 

LIGHT 


COMfEVTS 


*Utillaad  for  Total. CETA  Program  Only  , 
**0tili*td  tt  Activity  La%pl  only  % 


ERLC 
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3%6     MONTHLY  SERVICES  ASSESSMENT  FORM 


Z^VSE:     To. present  data  regarding  participant  services 
m    .      provided  through  PIC  program. 

i  , 

*  ^  REPORT  PERIOD 


to 


j  . 


INDICATORS 

Number  c 
'  Sex 

f  Clients 
ved 

INC  IE 
•  S£R 

VICE 

^ —  i 

/  TOTAL 
COSTS 

Staff 
.Time . 

.  * 

Plan 

Actual 

Var. 

'Warn. 

Man 

Act. 

Var. 

Plan 

Act.' 

^Var. 

Plan 

Act 

TOTAL  SERVICES  TO  fcLIENTS 

a* 

a — m 

• 

• 

• 

^  TOTAL  MANPOWER  SERVICES 

* 

.Intake  * 

r — 

.Orientation  ' 

* 

t  .Cotfhaelina/Coachtt* 

T       _#Job  Development 

.Placement  ^ 

-3 

c-    ^Placement  Follovup 

« 



 1 

* 

• 

> 

 :  — 

% 

TOTAL  .SUPPORT  SERVICES 

«     V  * 

1 — -te 
— ^ 

• 

.ChiTd  Care 

4 

*  Transportation 

t 

.Medical  Treatment 

;  '-  i 

i  

.Family  r*Lanning  . 

-  %       .Vocation  Rehab. 

1 

  %  

.Bonding 

t 

±   » 

* 

/  "  ■ 

Comparison 
Ratios  * 


Cost 
per 
Client 


Inci 

o^  Service 


^  Incident 


Staff  Time 


Percentage  of 
Total  Manpower 
Service  Costa 


Manpower  Services 


Supportive  Services. 


-   3.7     SAMPLE  TREND  ANALYSIS  MANAGEMENT  REPORTING 

DISPLAY  CHART  ,  - 

USE*  "  To  arrange  performance  >da,ta  in  such  a  fashion  that 

  it  is  easily  reviewable  for  management  control 

purposes. 


r 


UNHT:   WORK  EXPERIENCE 


OATE  OF  REPORT: 


OCT  NOV 


coRRtcTtvr  action  initiate 
♦     ,  TREND  ANALYSIS:  PERCENT  OF  PLANNED  ENROLLMENT 

UNIT-  WORK  EXPERIENCE  BY  CONTRACTOR  DATEjpf  REPORT  . 


v 


/ 

- 

^ 

»  f 

!  '  1 

!••••...'•• 

4  


V 


-4- 


OCT  NOV  OtC  JAN 

mmmm  CONTRACTOR  a 
■iii        i    CONTRACTOR  I  > 
•••••••  CONTRACTOR  C 


— r- 


MAM  APR  i        MAY  1u*l  JULT 

CORRfCT^«   ACTION  IM!TlA?eO 


Aub  S«*T 


9 

ER|C 


584 


7* 


IV- 8  5 


>,  •  DEVELOP  EVALUATION  PLAN 


PME  AIDS 


step 


Section  III 
Reference  Pages, 


1.     Establish  -Scope  and.  Nature  of>Evaluati,on ^ 

1.1-  "Type  of  Evaluation  Questions 
1 2    Outcome  Evaluation .  Formats  • 
1.-3     Relative  Effectiveness  Evaluation  ^ 

Checklist  *  ,       '  . 

-1.4     Impefct  Evaluation  Checklist  and  -m 
' Questions  ^ 


lfl-2-59-260 
HI-260-261 

III-261-271  - 

IIIr271-'279* 


2.     Assign  Evaluation  Responsibilities  and  ^ 
Establish  Procedures 

2i    Evaluation  Review 'Process  Ques^iong  "JljJs^SI- 

.   *2    Evaluation  Responsibilities  Questions  JII  285 

3      ■  UV.^t-  Evaluation-  Research  Design  •»* 

3.l' 'Evaluation  Plan  Development  Questions  vgJ^JjrSOl 
3*2    MO^el  Evaluation  Plan  Format    .  JUi 

4.     create  Research  and- Data  Gathering 

Instruments  r  '  * 

4.1  Bas,ic  Baseline  Data  Worksheet     "  rn.305-.3f06 
„   "Format  ' 

4.2  Evaluation  Research  Instruments  .  III_30^_311 

Checklist  -  -,  ... 


^7 
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585 


USE:  To  determine  the'primary  .type<s)  of  evaluation  your  .PIC 
~ '  will  emplpy  to  evaluate  its.  programs'.  * 


QUESTIONS 


Answer  the  questions  below  to  begin  t<*  determine  th^  general 
scppe  and  focus  of  .your'  PlC'  s  ? evaluation.     Then  det^nu^e 
what' relative  proportiQrv'of  your*  PIC's  overall  evali 
^effort*  should  be  concentrated  in  each  of  those  are^ 
/^ycjft  defined  as-,  important.  * 


itibn 
is  which 


Very 
Imp. 


Imp. 


Not- 
Imp. 


5v     '  4 


3        2'  1 


How  important  is  it,  at  this  point 
-  in  -its  evolution,  for  youtar  PIC  to  m 
'Jcnow  h<9*  well  your*  program  performed 

against  those  standards  set  at  the 
'  beginning  of  the  performance  period? 

(outcome  evaluation)*  . 

■»  •  * 

How  important*is *it  for  your  PlC 
to  be  able  to  assess  and  compare 
the  effectiveness  of  projects, 
^tactics,  service  deliverers,-  and 
program  components  (Relative 
Effective  Ev^uat^on) 

How  important  is  it  for  your  TIC 
to  be  able  to  determine  the  extent  v 
to  which"  your  programs  have  brought  '  * 

about  institutional  change  or 
achieved  your  PIC's  broad  goals? 

r( Impact  .Evaluation)  '  ;  *  . 

What  relative  proportion  of  youi*  PIC's  evaluation^ef forts 
should  it  allocate  to  the.  types-  of  evaluation  lisCftd  below? 


5  .4 


2  1 


•   •  Type 

Outcome  Evaluation , 
Relative  Effectiveness 

Evaluation 
Impact  Evaluation 


Proportion  of  Total  Evaluation  Efjprt 


IV-89  ;.5S(r 


ERIC 


1.3   •  OUTCOME  EVALUATION  FORMATS  V 

.  .     .  .  v. 

USE:*  To  determine  what  outdome  indicators  and, 
standards  will  be  used  to  evaluate  'your 
PIC's  program  suceesjs. 


~~ ~  777  •  .  

Refer  to  forms  under  Stage  B,  Step  1,  pp.  IV-17  to  IV-43. 


v.  4 

A  . 


4- 


r. 


5S7 
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1.3     RELATIVE  EFFECTIVENESS  EVALUATION 
CHECKLIST    >  m 


USE:    To  determine  what  types  of  effectiveness  indexes  to  use 
]     to  evaluate  your  PIC's  programs. 


„  < 


Checklists . 

There  are  three*  basic  forms  of  indexes  which  can  be  used  to 
assess  the  relative  effectiveness  of  your  PIC's  efforts. 
These  indexes  are  defined  below.     Check  the  type  of  index (es) 
which  your  PIC  will  use  to  compare  ef fectiverWs.s  and  write 
in  the  space  designated  the  specific  index  or  indicators  - 
that  you  want  to  apply. 


TYRES  OF  INDEXES 


TO  BE 
USED  (</) 


INDEX  MEASURE  (S). 
OR  INDICATORS  TO 
BE  APPLIED 


•'Effectiveness  -  performance  measure 
which  combines  two  or  more  measures 
of  effectiveness  to  derive  a  result 
stated  in  numerical  terms  (e.g., 
placement/wage  index) 


4 

Cost  Effectiveness  -  performance 
measures  which  combine  or 
associate  costs  With  effectiveness 
indexes  to  determine  the  relation- 
ship of  costs  to  effectiveness  in 
achieving  planned  results. 


^Composite  -  performance  measure 
compiled  of  several  factors  (e.g., 
planned  versus  actual  performance; 
extent  of  service  or  more  in  need; 
cost  effectiveness;  extent  to  which 
participants  complete  component; 
weighted  according  to  .their  impor- 
tance. 


IV- 91 
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ERIC 


1.4     IHyCT  EVALUATION  CHECKLIST  AND  QUESTIONS 

USE :  .  To fseleqt" those  areas  in  which  to  do  impact  evaluations 
..'  '    '"-snd.to  determine  the  general  type  of .  evaluation  to 
implements"  '  • 

•i  -  "  '  t  • 

*">  J   *  .  QUESTIONS  AND  CHECKLIST 

'  *  *  ■ 

There  are  three  general  ^reas  in- which  to  do  impact  evaluation, 
Check^-below  the  areas  in  which  you  wish  to  conduct  evaluation, 

_  Participant  Change 
X    •  '  Cost  benefits  . 

Community  or  institutional  change^  . 


Thjen,  based  oiv  the  .area  ( s)  which  you  have  selected ,  determine 
the  cjeneral  typ^  'of  evaluation  which  you  wish  to  do  and/or  . 
the  qii^tions  you  wish  to  .be  able  to  answer.  ' 

Participant  Change-  Evaluation 

<  ^     "  *•  "  " 

Wha'V  flMlsufes  or  indicators   (£.g.  ,  earnings  gain,  private 
sectorSfbbs  secured,  length  of,  employmeat)  do  you  want  to 
lctok  ^t^tg^  assess  participant  change?    List  these  measures 
b£lqw.  i" 


What  method (s)  will  you  employ  to  :apply  these  measures? 
Check*         the  appropriate  'technique  from  among  those 
listed  keloy.  .  - 

z    -    Follow-up  Studies  + 
■  ,    Time  Series  Design 


^Comparison  and/or  Control  Group 
Cost  Benefits  Evaluation 

If  you  have  selected  the  cost  benefit  method  of  evaluation, 
what  techniques  do  you  intend  to  employ?    Check  the  appropriate" 
technique (s)'  listed  below.  "  « 

'  net  present  value  approach 

*   benefit-cost  ratio 

,>r    ,  internal  rate  of  return  method 

payback  method  » 


Copgnunity  or  Institutional  Change  .  /w 

*  *  \  1  v 

If  you  have  dedlded  to  look  at  community  or  institutional  change, 
check  o%£r  from  the  areas  and  the  questions  listed  below  those 
which  will  become  the  focus  for  your  inquiry. 

IV-92 


AREA  FOJf 
EVALUATION    [i/ ), 


io 


QUESTIONS 

ASK  (y) 


\ 


Institutional  Impact  of  Title  VII. 

.  Has, the  P^C  stimulated  new  business  * 

;   participation  in  iocal  employment 

/   £nd  training  efforts? 

* 

Has  the  PIC  stimulated  new  responsive- 
ness on  the  part  of  "traditioriaL" 
'   employment  and  training  deliverers 
•to  the  needs  of  business? 

r 

.  Is  ther§  a  comprehensive  planning 
approach  between  PIC  and  the  local 
Employment  and  Training  Council? 

.  Has  the  Pic/stimulated  any  -areawide  r 
planning  between  different  prime 
sponsorship'  areas  and  other  PICs? 

/  Other 


Attitudinal  Impact  of  Title  VII  ,  1 

.  How  has^  the  PIC  program  affected  the 
attiti^es  of  employers  toward  the 
program,  toward  employment  and/ training 
programs  in  general;  and  towa/d  eligible 
.clients-  as  potential  employees? 

.  How  has  the  PIC  program  impacted 
on  the  attibudes  and  motivations 
of  participant's?  4 

.  Other  r 

/' 

Patterns  of  Service  / 

.  What  are  the  characteristics  of  the 
'  patticipants  being  served?  ,  How  dO 
they*  compare  to  regular  CETA  clients, 
goals*  and  some  statement  of  "universe 
of  ae£d"? 

.  What  are  the  characteristics  of  the 

businesses  being,  served?  Wheffe  do  they 
.  stand  in  the  overall  eoonomic  structure 
'  *0f  the  area?  . 

Other 


Cost 


iJEiat  reasonable  cost  'calculation^  "can  be  - 
made  about  some  of  the  above  impacts?   

Other 
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2.1     EVALUATION  REVIEW  PROCESS  QUESTIONS 


USE:     To  determine  your  PIC's  basic  process* for  evaluating 
program  results.  .  *  x*  '  % 


QUESTIONS 


Answe 


the  following  questions:  * 

What  is  the  overall  purpose  of  your  PIC's  * 
evaluation?  .  *  \     y  \ ' 

*  « 
What  are^ the  expectations  regarding  the' 
evaluation?    What  should  it  enable  you  to 
tell  „or  do?  .  /f 

What  resources- will  be  expended  for  the* 
evaluation?  ^ 

What- methods  will  be  used  to  coordinate  the 
PIC's  evaluation  activities  with  those  .being 
undertaken  by  the  Prime  Spohsor? 

What  procedures  will  be  devised  for  the 
systematic  review  of  evaluation  plans, 
methodologies,  (ifcta,  and  findings? 

What  will  be  the  due  dates  fo,r  any  "report's  , 
'  or 'documents  from  the  evaluation? 
*  *  * 

What  methods  will- be  used  for  monitoring  the 
evaluation  (e.g.,  monthly  meetings  or  progress 
reports) ? 

What  will  be  the  policies  regarding  the 
t  retention  bf  records 'and  raw 'data? 

What  procedures  will  be  used  for  resolving 
problem  issues  as  they  occur? 


9 

ERIC 
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1 


2.2     EVALUATION  RESPONSIBILITIES  QUESTIONS 


USE:     To  assign  responsibilities  for  the  conduct  of  your  PIC's 
evaluation.- 


QUESTIONS 


In  order 
conduct 


^^ITterm^i^fho  should  be  responsible  far  the  actual 
of  yov^r  PIC's  evaluation,  answer  the  following  questions: 

How  much  emphasis  is  being  placed  on  the 
evaluation  at  your  PIC?        *  , 

What  resources  will  be  ,made  available  to  4 
implement  your  evaluation  plan? 

What 'will  the  role"  of  the  Council  be  in  the 
evaluation?  ......  1 

Who  will  have  -the  ultimate  responsibility 
-for  administering  the  evaluation  program? 

Who  will  supervise  the  evaluator (s) ? 

,  .  Who  wiii  be  involved  in  the  analysis  of  <* 
fevaluation  reports  and  an  the  interpretation 
,   of  data? 

t 

To  whom  will  those  individuals^  charged  with 
conducting  the  evaluation  report? 

%What  will  the  responsibilities  of  staff 'be 


in  the/ evaluation? 


■  » 
Do  anyV-eurrent  staff  members  have  th3 
capability  (or  the  time) „to  perform 
evaluation  research? 

Wbuld'it  make  more  sense  to  hire  a  full-time 
evaluation  specialist  or  employ  private      .   ^  ^ 
contractors  on  anx  ad  hoc  basis? 


9 
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3.1/  -EVALUATION  PLAN  DEVELOPMENT  QUESTIONS 


USE  j 


To  determine. the  basic  structure,  content,  anS  format  > 
for  your  PIC"  s  evaluation  program  for  tfrii^fiscal  year. 


QUESTIONS 


Answer  the  following  questions: 


Why  is  the  PIC's  program  being  evaluated? 

What  are  vthe  primary  and  secondary  objectives 
(standards) *  of  the  program?    x  * 

V(Jiich  of  these  objectives   (standards)  are 
stated  in  readily  measurable  terms? 

f 

What  are  t^e  hypotheses  you  want  to 'test? 

r 

What  resources  are  -available  to  evaluate 
the  program? 

How  much  timd  is  permitted-  for  the  evaluatioh? 

What  data  is  needed? 

» 

%  « 

Where  does  this  data  'exist? 

How  can  this  data  be  obtained?'  -  ,  * 

How  will  this  data  be  analyzed?  ♦ 


How  will  the  findings  be  tflHed? 

What  format  will  be; best  for  presenting 
your  evaluation  -findings? 


59:i 


> 
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3. '2     MODEL  EVALUATJON1  PLAN  FORMAT" 

A 

USE:    To- present  sunupary  inf  ormati&jf  «rejgardijig  your  PIC'^ 
~~      evaluation  plan  for  the  fiscal  ^ear. 


Your  PIC's  evaluation  plan  should  ^consist  of  the  fallowing 

sections:  * 

.  V  * 
"  '1.  \    Purggse :     states  the "reasons  why  the  study  is 

Being  undertaken.  .  Incorporates  a  brief  descrip- 
tion of  the  program  and  its  goals. 


2. 


3. 


4. 


5. 


7. 


Rationale:     states*  succinctly  reasons  for,  and 
purposes  of,  evaluation,  and  how  study  results 
,  will  be  used,. 


Objectives : '  specifies  objectives  and  scope  for 
the  evaluation.. 


nect: 


L 


progress  of  the  evaluation, 
deadlines  and  due  dates. 


Highlight^ all 


r  \ 


^thpdology :     the  heart  of  the  research  design, 
states  exactly  how  the  evaluation  will  be  performed. 
Details:        .  ^ 

data  that  will  be  needed  , 
'        .     techniquesof  data  collection 

.     means  of  data  aggregation  and  ^nalysi^.  / 

-Instruments:     describes  the  types  of  instruments 
and  worksheets  needed  to  collect  and  treat  the 
data  and  discusses  the  process  by  whrqh  they 
will  be  developed*  1 

6.    •  Schedule:     sets  out  a  timetable  to  govei^i  the 


Reports :     indicates  th£i_types  or  .reports/ that 
will  result  from  the  evaluation,  including  ' 
their  content,  format, /and  submission  dates. 


A 


A 
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4.1     BASIC  BASELINE  DATA,  WORKSHEET 

USfi^   To  analyze  performance ] Against  plan  employing 
<Jata  retrieved  through  your  PIC's  reporting 
system.  * 


* 

• 

• 

* 

Project/ 
Area 

Indicator 
—  1-  u 

Standard 
or  Plan 

Actual 
Performance 

Variance' 
(+  %) 
>  

• 

• 

* 

• 

t 

• 

*  ?> 

* 

♦ 

O 

♦ 

*  • 

■ 

i  ' 

• 

'    IV- 9 8 
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4.2     EVALUATION  RESEARCH  INSTRUMENT  CHECKLIST 

USE:    To  check  the  effectiveness  of  your  evaluation  research" 
instruments. 


fcIf  your  evaluation  research  instrument  is  effective,  you  shoiild 
be  able  to  .answer  "yes"  to.  the  following  questions: 


J    Is  the-  data  collection  effort  restricted 
to  that  information  required  to  achieve 
the  objectives  df  the  evaluation?  • 

Have  you  ensured  that  your  instrument  is 
comprehensive  enough  in  scope  to  answer 
all  questions  which  are  importapt  to  your 
evaluation  effort? 

Is  the  wording  of  items  simple  and  non- 
technical so  that  they  can  be  easily 
interpreted  by  re3pondents? 

If  technical  terms  must  be  used,  have 
.they'ibeen  defined  so  that  they  can  be  t 
understood  uniformly  by  both  the 
respondent  an^  the  data  collector? 

Have  the  instruments  been  designed  so 
that  they  4fG  streamlined  in  format  and. 
easy  to  use?  - 


Yes 
□ 


Yes 

a 


Yes- 
D 


Yes 


Yes 
□ 


NO 
□ 


NO 
□ 


NO 

D 


.  NO 
□ 


NO 
P 


* 


ERJC 


IV- 9  9 


6,  •  MONITOR  PROGRAM 


.    -  "  .  PME  AIDS        .  , 

Section  III 

ste{>  Reference  Pages 

  .  "  N 

1,    Analyze  Planned  Versus  Actual  - 
Performance  " 

1.1  Basic  Participant  -Information  on  ' 
Monthly  Reports  lll'lil 

1.2  Participant  Monthly  Report  Format  J"  „J  „n 

1.3  EGS  Monthly  Report  Format  Ilj-^u-^i 

1.4  Performance  Versus  Plan  Control 

Board  Format  "J  l^f. 

1.5  Sample  On-site  Monitoring  Checklist 

1.6  On-site  Records  Review  Instruments  Ill-Ji/-^* 
— Employability  Development  Planning  ^ 

Occupational  Classroom  Training 
*      — OJT  ' 


r 


2.  Initiative  Corrective  Monitoring  .   j 

2.1  '  Corrective  Monitgring  Visit 

Checklist  .  ni-^/-4« 

4. 

3.  Ensure  Necessary  Corrective,  Actions 

'    3vl    Corrective  Action  Plan  Format  ,  111-33^-336 


/ 

1.1 


BASIC  PARTICIPANT  JNFORMATION  ON  MONTHLY  REPORTS 
CHECKLIST  ,  !  S  »  


OSE": 


To  ensure  monthly  reports -for  your  PIC  which  provide- 
the  essential  information  for  effective  program  r 
\  monitoring.  '  *• 

,     '  J 

«    I  •  • 

Items  that  should  be  included  on  your  participant-relate^ 
iponthly  reports  include:  - 

U         On  Board;     number  of  persons  'who  are  enrolled  in  the 

program  at  the  beginning  of  the  montjj-  , 

;   Entrifes  t(new  participants):    -number  of  persons  Vho 

enroll  in  the  program  during  the  month. 

  Exits   (terminations):     number  of  persons  who  cease 

to  be  enrolled  in  the  program  during  the  month  for 
any  reason,  including: 

 ^  Placements  -  persons*  who  enter  employment 

during  the* month  or  "who  enter  an  initial 
Employment  period.  ** 

  b.     Other  positive  terminations  -  persons  who 

transfer  to  other  training,  return  to^  school,, 
enter  the  military,  or  go  on.  to  other 
unsubsidized  employment.  * 

  c.     Nonpositive  terminations      persons  who  left 

'  the  program  fpr^  reasons  other  than  (a)  and 
(b)   above.  . 

  d.     Completers  in  holding  —  persons  who  complete 

the  program  during  the  month;  and  are*  waiting 
for  an  appropriate  job  opening.  * 

Othq^r  items  frequently  on  monthly  reports  include: 

  Participant  characteristics  ^ 

 _    Financial  measurements   (accrued  expenditures;  portion 

of  expenditures  transferred'  directly  to  participants). 


An  example  of  a  ..monthly  report  format  , follows  ,4 


7 
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1,2     PARTICIPANT  MONTHLY  REPORT  'FORMAT ' 

USE*:  Tb  collect  data  regarding  ^he  'performance  of 
" —      individual  participant-serving  projects.. 


MONTHLY  "PROGRAM/ ACTIVITY  REPORT 


Sponsor  ,__  .  ,  

Project  

Report  Period         '   , 

1.    Total  costs  (2,  3,  4',  and  5) 


2. 
3. 
4. 
5. 
Sy. 


Training  costs   

Administrative  costs 
Supportive  services 
Allowance  costs 


8. 
9. 
10. 


Total  cumulative  participants^ (6a  +  6b) 

a.    On-board  beginning  of  grant   

Hew  participants 


Total  terminations  (7a,  7b  +  ,7c)   

a.    Total  placei^fRts  (7a(l)  +  7a(2)) 

(1)  Direct  

'  (2) 


Indirect 


Total  tither  positive  terminations 

(1)    Other  vocational  training   

Moved  to  other  employment   

Back  to  school   

Military  » 

Other" 


(2) 
(3) 
(4) 
(5) 


x.    Total  negative  terminations 


(1) 
(2) 
(3) 
(4). 
(5) 
(6) 
(7) 
(8) 
(9) 
(10) 


Laid  off 


Health  —  pregnancy 
Family  care 


Moved  from  area  _.  

Refused  to  continue  i  

Administrative  separation 
Cannot  locate  . 

Incarcerated  .   

Married   n 

Other' 


TV* 


Current  enrollment  (7  -  6)   

Number  completing  activity  

Number  competing  Job  entry  period 

a.    Short-term  placements   . 

Mid-term  placements 


11.  4b 


b, 
c. 


Long-term  placements 
retention  of 
Short-term  placements 
Mid-term  placement*  _ 
Ldng-term  placements  _ 
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1.3     EGS  MONTHLY  REPORT  FORMAT 


USE:    To  coTTSb^data  froir^IGS  projects, 


REPORT  FORMAT 


Planned 


Products/Deliverables  Expecte4 
Activities/Tasks  Projected 

Results  Expected-  ' 

if  ..»♦-. 
Projected,  Expenditure?  * 


Actual 

Products  Produced 

Activities/Tasks 
Completed 

Results  Achieved 

Actual  Expenditures 


.  •  Significant  Problems  Endoufttered 
Proposed^ Modi f ications  to  Project  / 


0 


A. 


J*. 
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USE: 


1.4     PERFORMANCE  VERSUS  PLAN  CONTROL  BOARD  FORMAT 

To  summarize  and  present  graphically  data  from 
individual  project  ^reports  in  such  a  way  that  it 
-can  «be  used  to  monitor  projSfct,  component  and 
program  performance. 


*  / 
/ 
/ 


% 

100 
90 
80 
70 
60 
50 
40 
30 
.20 
10 
0 


+  15% 

-                                                                     Planned  laval 

\  ^Cumulative 

-15% 

\Monthly 

✓      -«*  ♦ 

Ptacamant 

July  Au\j.   Sapt.  Oct.   Nov.   Deo.  Jan.   Fab.   Mar.   April   May  Juna 


PERFORMANCE 
CHART 


/ 

» 

\ 

/ 

\ 

/ 
/ 
* 

\ 

DATA 

TABLE, 

\ 

-i  * 

Ctaaaroom  (aklfl  training) 

-Plan 

Actual 

Common  Data  Ham 

July 

Aug. 

Sapt. 

Oct. 

Nov.       Dac.  . 

To  Data 

to  Data 

.In  procaaa  bagjnnlng  month 
Entrloa  during  month 

490 
91 

497 
92 

516 

*  119 

■i 

53b  563 
111 

550.0  av.  ' 
85.0  av. 

52Q.5  av. 
*10o!4  av. 

Extta  during  month 

M 

73 

106 

66 

78 

66.0  av. 

65.6  av. 

*tateamant  during  month 

•7  . 

60 

65 

46 

26 

63.6  av. 

53.8  av. 

Haoamant  rata  monthly  ( % ) 

79.8 

62.2 

60.2 

55.8 

#35.9 

75.0 

62.6  av. 

*acornant  rata  cumulate*  (%) 

79  J 

60.9 

72.5 

*  66.4 

*62.5 

750 

62.5  cum; 
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1.5     SAMPLE  ON-SITE  MONITORING  CHECKLIST 

r   :  

USE':     To  revieW  various  aspects  of  participant  serving 
^  projects.  ,  ' 

The  condensed  checklist  wlv^ch  follows  was  developed  from  a 
25-page  document  covering  a  wide  range  of  program  services, 
administration,  and  financial  control  categories. 


Data 


AREA:     OUTREACH  AND  RECRUITMENT 


1.      How  many  individuals  have  been  recruited  for 
programs?  m 

2*      By  whom  are  they  recruited?  (Agency?) 

3.  How  many  of  the  'total  recruited  have  been 
determined  to  be  ineligible  for,  program  or 
services? 

4.  What  are  reasons  for  ineligibility  of  those 
recruited  during  this  period?  .  • 

5.  What  is  the  age  and  sex  breakout  of  in4ividuals 
screened?  ! 

Male  Under  22          ^22-44  45+  

Femarle        Under  22  22-44  45+  


6.      How  many  are  disadvantaged? 


7.      How  many  are  members  ©f.  a  minority  groyp?   

Process 


^      What  are  sources  of  recruitment?     (Agency  referrals? 
Outreach?  Walk-in?)  ^ 

2.  Wfiat  4ype  of  community  agencies,  employment  services 
welfare  departments ,  service-  clubs,  court  systems, 
and  'schools  are  being  used? 

3.  How  many  people  seek  assistance*  a  week? 

4.  '  In  terms  of  numbers ,  are  too  many  or  too  few  people 

being  reached?  • 

5.  Is  a  problem'  developing  because  the  wrong  people 
(target  groups)  are  being  recruited? 
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AREA;    'ON-THE-JOB  TRAINING  SERVICES 


flow  many  people  were  enrolled  in  the  program  during 
this  time  frame?   

List;  the*  occupations  in  which  training  is  being 
conducted  along  with  starting  wage  and  wages  at 
completion  of  training. 

Wage  at  Completion 
Occupation        Starting  Wage  ,         ,.,of  Training   


What  is  the  average  length  cf  training?   

What  is  the  age  and  sex  breakout  of  people 
receiving  on-the-job  training  services?  » 

Male         Under  22        22-44  45+  

Female      Under  22         22-44    45+  

Number  of  disadvantaged?   Minority?   r 

Explain  the  career  ladders  (i.e.,  stages  of 
orderly  progression  which  result  in  long-range 
advancement) . 

i 

Are  private , employers  involved  in  the  program,  along 
with  gublic  and  nonprofit  agencies? 

Areothere  occupations  in  the  area  that  need  people 
trained  in  certain  skills?    Can  }ob  opportunities 
be  developed  after  people  are  trained? 

Is  the* business  world. receptive  to  manpower  training 
and^on-the-job  trainees? 

What  provisions  are  made  for  retention  tff  on-the-job. 
enr©llees  by /employer  afterS^he  program  Lb  completed? 


603 


AREA:     SKILLS ^TRAINING 

Data         f  ' 

1*^    HOw  many  individuals  are  enrolled  by  particular 
-y      occupation?     (Include  name  of  occupational  field.) 

s       2i      What  is  the  length  of  the  training  period?*  J 

3.  '     What' is  the  age  and  sex  breakout  of  the  enrol\fees? 

Male         Under  22           22-44  4*5+  

Female      Under  22    22-44    45+   •  • 

•  * 

4.  Number  of  disadvantaged?    Minority?  ' 


Process  j)t 


ata 


1.  What  opportunities  for  skills  training  exist?  Who 
runs  them?    What  is  the  enrollment  process?  Does 
the  sponsor  have  any  speciaj.  privileges  for  enrolling 
applicants?  .  • 

s  t 

2.  \   What  jobs  are  being  trained  for?    Wage  levels?  How 

appropriate  are  the  jobs  to  the  labor,  market? 

3.  What  is  the  structure  of  the  program-:- is" it  se£  up 

in  single  occupation  groups,  broad  occupation/  clusters ^ 
or  purchased  singly  for  an  individual? 

< 

4.  Are  projects  operated  on  a  specified  schedule  br  on 
an  open-entry/open-exit  basis? 

5.  Are  the  times  for  entering  training  flexible? 

*  .  i 

s. 

AREA:     ACCOUNTING  AND  FINANCIAL  CONTROL  ] 


1.  Does  the  accounting  system  identify  the  receipt  and  . 
expenditures  of  funds  separated  for  each  program?  > 

2.  Does  £he  accounting  system  provide  for  adequate 
budgetary  control  to  preclude  incurring  obligations 
in  excess-**  tWfcal  funds  ayailable  or  other  contract 
limitations?  *  | 

•  \/ 

3.  (  Is  the  responsibility  over  accounting  records 

•    separated  from  the  handling  of  the  sponsor's  cash  • 
receipted" or  other  disbursements?  _ 

4.  the  general  ledger  maintained  in  ink,  with  a 
\  chart  of  accounts  based  on  the  contract  budget- 
*  monthly  accumulative  and  balance? 


5.' ,     Were  funds  advanced  by  the  £rime  Sponsor  deposited 

accoun 
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in  a  special  bank  account? 
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^USE: 


"1.6     ON-SITE  RECORDS  REVIEW  INSTRUMENTS  .  - 

To  review  project  or  contractor  data  to  ensure  conformance 
to  plan  and  correctness  of  data  which  is  reported. 


i  ONSITE  RECORDS  REVIEW  INSTRUMENT 


t 


compontent  , 
or  activity  _ 

Service  provider 


EMPLOYABU-ITY.  DEVELOPMENT  PLANNING 

PERIOD  COVERED  BY  REVIEW. 
:   FROM:  TO  


number  of 
records'reviewcd; 

date  of  review. 


9 

ERIC 


AREA3  TO  BE  MONITORED 


ASSIGNMENT  STANDARDS  FOR 
COMPONENT  Rf  FtRRED  TO  MET 


COUNSELOR  A8SIONED  ON  FIR8Tr 
DAY  AFTER  REFERRAL 


COUNSELING  LOGS  SHOW  WEEKLY 
CONTACT  WITH  PARTICIPANT 


tDP  CONTAINS: 
-  GOAL 


OBJECTIVES  AND  8TEP8 


-  ASSESSMENT  DATA\ 


-  CHARACTERISTICS 

-  DOCUMENTATION  THAT  ANY 
NEEDED  SUPPORTIVE  SERVICED 
PROVIDED 


W  A  COMPONENT  ACTIVITY  HA8  , 
BEEN  COMPLETED  EDP  HAS  BEEN 
UPDATED  I  


TALLY: 

YES 

NO 

V 

STANDARD 

PERCENT 
MEETING 
STANDARD 

• 

FINDINGS  AND  RECOMMENDATIONS  < 

100% 

>  * 

100%  * 

/■ 

100% 

100% 

 *  

—   1 

•  i 

*  100% 

60! 
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ONSITE  RECORDS  REVIEW  INSTRUMENT 


COMPONENT 

OR' ACTIVITY  occupational  classroom  training 


SERVICE  PROVIDER 


PERIOD 
COVERED  BY  REVIEW: 

FROM  TO  


NUMBER  OF  - 
RECORDS  REVIEWED 

DATE  OK  REVIEW  . 

/ 


< 
i 


AREAS  TO  DC  MUNUUncu 

tally: 

STANDARD 

PEpCENT 
MEETING 
STANDARD 

FINDINGS  AND  RECOMMENDATIONS 

YES 

NO 

PARTICIPANTS  COMPLETING 

"»  

COMPETENCIES  IN  EDP  MET 

* 

96% 

r 

* 

OBTAINED  TRAINING  RELATECT 
UN8UB8IDIZED  EMPLOYMENT  . 

'   t 

76%  A 

PARTICIPANTS  TERMINATED: 
POSITIVELY  COMPLETED 

86% 

i 

> 



NONPOSITIVELY  TERMINATED 
PRIOR  TO  COMPLETION 

V 

> 

t 

10% 
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Form  or  Document:    In  04^ Training  Record 


Goals 


Data  Items  on  Form 


Results 


2 

i 

H 
H 


All  enrollees  shall-participate  Attendance  records 
in  motivational  training  .  • 

•  J- 

Enrollees  shall  be  counseled 
after  unexcused  absence 


Overall  group  attendance  at 
sessions  shall  be  90%         ,  ' 

Job  interviews  shall  be 
arranged  for  participants 
within  one  week  after  completion 
of  motivational  training 

Jobs  shall  be  developed  which 
pay -ft  least  $3. 15, per  hour  t 


Attendance  records, 
counselor  fil^ps 

Week  ly  *a  1 1 edanc  e 
*  records 

Job  developer  records 


Job  developer  records 


 %  in  Motivational  tjr^iniog 

 %  of  unexcused  absences  , 

counseled  ( 

 7.  attending 


7m  of  enrollees  whose  first 
Interview  is  within  ene  week 
bf  motivational  training 


X  jobs  developed  at  or 
aFove  $3.15  per  hour 


6K< 


erJc 
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USE*;    To  establish  an  effective  process  for  corrective 

monitoring  by  your  PIC.                                             .  * 

• 

QUESTIONS 

1.      How  will  your  monthly  performance  versus  plan  data  be 
reviewed?                                   %  • 

* 

« 

2.      Who  will  participate  in  that  review? 

* 

* 

■■                                    •  » 

3,      What  control  limits  will  be  set  on  your  PIC's  projects? 
»  - 

• 

* 

— 1 — : — : —  

4.      How  will  your  PlC'decide;  which  projects  need  corrective 

monitoring  visits?  * 

> 

 i— — 

5.      Who  will  be  assigned  to  make  those  units? 

:  ^  

6.-    /flow  will  the  project  be  notified  of  the  visit? 

i 

» 
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7.      What  process  will  be  empioyed  to  conduct  the  visit'? 


v 


8.  1     Who  will  receive. the  correct  monitoring  visit  ^Bort? 


9.      -How  will  the  project  be  allowed  to  participate  in  shaping 
•or  respond  to  the  corrective  monitoring  visit  reportZ 


Rir 


IV-lU 
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3.1   -CORRECTIVE  ACTION  PLAN  FORMAT  ■  - 

■  "  ~  \ 

o 

(  4 

USE:    To  present  tlje  results  and  determination  o%  your 
corrective  action  visits  in  "a  uniform  and 
consistent  manner. 


Format  $ 

*  •  \ 

Your  PIC'fr  corrective  action*  plan  for  each  visit  should  . 
address-  eaqh  area* where  variance  from' plan  was  significant 
For  each  area,  the  plan  should  specify  the: 

— problem  and  its  source 
— extent  of  deviation  from  the  plan 
— criteria  required  for  improvements 
--remedial,  steps  to  be  taken 
— time  frame  for*  improvement 
y    — assignment  of  responsibility  for  action 
— special  monitoring  arrangements  to  be 
$    implemented*  t 


H.   EVALUATE  RESULTS 


PME  AIDS 


Section  III 
Reference  P*ages 


Collect  Data 

1.1    Follow-up  S<fudy  Questions 
1^   Follow-up  Study  Project  Report 

Format     '  .        '  " 
1V3    Follow-up  Study  Comparative 
Analysis'  Format  ^> 

Analyze  Data 

2.1  Analytical  Techniques'-  refer 

to  Stage  B,  Step  1,  pp.  ?""g^:3 
and  Stage  E,  Step  3,  pp ^IJI-^Jjf 253 

2.2  Goals  Measurement  Sheet 

Feedback-  into  Planning-  Process^ 
'  3  l    Evaluation  Report  Formal 


III-.349:353 
III-353-.363 
III-361-362 

in-518  .  . 

o 

III-367-369 
III-373-379 
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1.1    FOLLOW- OP  STUDY  QUESTIONS 


USE:    To  determine  nature  and  'extent  of  follow-up  study, 


QUESTIONS 


-Answer  these  questions  to  determine  the  structure  of  your  PIT's 
foliow-up  study.  *  ,  0 

. 1.      What  information  should  be  collected?    Is  any" of  the 
following  data  important  to  your  study? 

*  • 

Evaluation  management  data; 

'7  Respondent  identififers;  i.e.-,  name,-  Social 
.  .  Security  number,  and  so  forth 

.  Respondent  locator;  i.e.,  address,  tele- 
phone number,  a  contact  person,,  and  so 
"forth     m  - 

.  Type  or  method  of  contact;  i.e. ,  mail, 
telephone,  personal,  and  so  for^h 

Demographic  attributes  and  variables: 

\  Standard;  i.e.,  age,  sex,  and  ethnicity 

.  Household  responsibilities;  'i.e. ,  marital 
status,  head  of  hou-ehold,  dependents, 
and  so  on 

/.Special  group  identifiers;  i.e.,  public 

assistanqe  status,  veteran  jstartus/  handi-  A 
capped,  and  so  on 

.Program  and  service  descriptors; 
.  Major  program;   i.e.,  on-the-job  training, 
cl'assroom,  basic  education,  public  service 
education,  and  so  on 

.  Services,  received;  i.e.>  component 
descriptors  *  * 


Length  of  services 
units  and  so  ori. 


i.e.,  time,  service 


\  Service 'provider  identifiers 

flork  history  description  . 
For  pre-program  and  post-program  periods, 
chronological  lists  and  descriptions  of 
periods  ^of : 
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Employment. . .employed  period  descriptions 
including: 

-  Job  finding  activities ...  friends, 
.  want  ads,  door  pounding,  public 

agency,  private  agency,  and  so  forth 

-  Employer  identifiers ...  name,  address, 
telephone  number,  standard  industrial 
classification  (SIC)  code,  and  so  forth 

-  Job  descriptors. . .Dictionary  of  Occu-  f 
pational  Titles   (DOT)  dr  other*,  '      "  . 
unionized,  skill  level,  and  so  forth  - 

-  Wage  alid  benefit  quantifiers. .  .dollar 
values 

-  }iotk  intensity. .  .full  or  part-time 
hours  worked,  and  so  forth  W 

-  Perception  of  supervisor 

-  indicators  of  internal  mobility .. .wage 
change,  job  change,  promotion,  and 

so  on 

-  Deration  of  employment 

-  Reason  for  termination  if  applicable 

:    *       Unemployment/,  .period  descriptions  deluding-: 

»  -  Job  finding  activities ...  friends, 

want  ads,  door  pounding,  publicv 
agency,  private  agency,  ah4  sq  forth 

-  Sources  of  income  during  unemployed 
period  , 

-  Duration  of  spell  of  unemployment 

Not  in  labor  force. . .period  descriptions  including; 

-  Activity  while  not  in  the  labor  force... 
in  school,  military,  pregnant,  house- 
hold responsibilities,  discouraged, 
prison,  and  so  forth  w 

-  Intention  to  participate  in  labor 
force  . 


\ 


Income  descriptors; 
\    •  Squrcfes  of  Income;  i.'e.,  w^ges  and- salaries, 
publilc  assistance,  unemployment  compensation, 
and  s©  on 

•  AmouftC  of  income  % 

*-  a 

Attitudinal  and  behavioral  indicators; 
.  Perception*  of  job  obtained 

.  Perception  of  training  received 

.  Recommendations;  i.e.,  services  quality, 
contn^,  aod  b9  on 

.  Indicators  of  antisocial  behavior;  i.e., 
arrest  record,  drug  use,  and  so  on 

. .  Knowledge  of  manpower  services  available 
^    '^o  participant 

"  Indicatory  oV  employers'  perception  of  clients i 
.  Conformance  to  work  discipline  

.  Productivity  s  • 

.  .  Sociarl  ^cceptanceiin  the  worjj  environment 


\ 


■» 
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Who  should  be  contacted? 

Proportion  of  clients  to  follow? 
.    ;Size  of  comparison  group? 
Sample  size? 


How  long  should  follow-up  intervals  be? 

.     30,  60,  90, .180,  365  days? 

in  correspondence  with  national',  state 
or  local  studies? 


How  will  you  follow  up? 


Group  interviews? 
individual  interviews?. 
Personal  interviews? 
•Telephone  interviews? 
Mail  surveys? 


'1.2     FOLLOW-UP  STUDY  PROJECT  REPORT  FORMAT  • 

t 

USE:     To  present  the  results  of  £  follow-up  study  on  a, 
particular  project.  " 


pro  31 

Summary  Project  Report 


Name  of  project:   

Date,  of  beginning  of  project 


Date 


Date  of  ending 


Average  length  of  project  per  completer:  :  

Service/activities  in  basic  project:*  * 

No.  enrolled:  ,   No.  Completed:  ^ 


Av«  Mo.  Earning 


No.  placed  within  3  months*  of  completion:.   

No.  employed  at  3  month  follow-up:  ^  Av.  Mo.  Earnings 


No.  employed  after  6  months  in  occupation  for  which  developed: * 


Tatal  cost: 


Average  cost  per  enrollee : 


Average  cost  per  completer: 


Average  cost  per  completer  employed  after  6  months: 


Lis,t  of  activities/services  in  program  ^nd  contractor  or  subagent 
for  each: 


Acttivities/SerVices 


Contractor  of  Subagent 


 *  *            -   :  :  

 L  » 

- — \ 

 \ 

 :  *  

\  ' 

Othe^  pert 

bnent  infoination  on  quality,  cost,  3if  f  icultie's ,  etc.: 

 ;  «  «  .  

 .  .  ,  .  *  , 

Other  materials  on  quality  including  evaluation  apalyses: 
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1  3     FOLLOW-UP  STUDY  COMPARATIVE  ANALYSIS  FORMAT 

•  ,  — ■  T  ~~ 

USE:  To  display  results  of  a  %fol low-up % study  on  a  number 
~ —     of  projects  or  contractors. 


LABOR  FORCE  STATUS  BY 
OJT  CONTRACTER  t 
DEMOGRAPHIC  GROUP 


CONTRACT  PERIO0 

FROM  -TO  

REPORTING  PERIOD 
FROM  JO  


CONlRACTER  * 

/oemoorafhk;  dftou* 


CONTRACTER  * 


-  MMOItlTY  «ALE 


.  -  WHITE  MALE 


MIMORITY  FEMALE 


WHlTE  FEMALE 


contracted  • 


-  MMO«rrr  male 


%  WHITE  MAU 


UWOMTT  FEMALE 


WHITE  FEMALE 


CONTRACTOR  C 


.  IMKOftfTY  MALE 


-  WHITE  MALE 


.  IMMOAfTY  FEMALE 


-  WHITE  FEMALE 


CONTRACTS*  0 


HNtORfTY  MALE 


•  WHITS  MALE 


.  iMMOHrTY  FEMALE 


-  WHITE  FEMALE 


CONTRACTER  E 


-  MWtOftfTY  MALI 


WHITE  WAU 


MINORITY  FEMALE 


-"WHITE  FEMALE 


„    |  MONTH 

•  MONTHS  « 

12  WONTHt , 

RE  TSNTtONE 

0 

~j  m 
<  W 

RETENTION* 

w  • 

w  O 

o 

£  2  - 

RETENTK) 

Nft 

1  0 
_  n 

O  w 

VUMOER 

>-  « 
o  -* 
p. 

NUMIER 

o  5 

^  A. 

HUMEER 

• 

r 

 1 

— ^  

f 

r 

> 

I 

« 

► 

f 
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2.1    ANALYTICAL  TECHNIQUES 

USE:    To  determine  What  analytical  techniques 
and  methods  to  apply  to  examine  youj: 
PIC's  evaluation  data* 


Refer  to  material  and 

forms  under 

Stage 

B,' 

Stiep  1,  pp.  111-65-81 

and  Stage  E 

,  Step 

3, 

pp.  111-243-253^ 

> 

♦ 

021 
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2.2     OBJECTIVES /MEASUREMENT  ASSESSMENT 'SHEET 

« 

USE:    To  assess' the  results  of  a  project  at  its  completion 
~~      and  six  months  'thereafter  in  terms  of  its 
accomplishment  of  objectives. 


r  •       ^  '  - 

OBJECTIVES  MEASUREMENT  EXAMPLE  WORKSHEET 

FOR  ' 
OJT  PROGRAM  AT  CAREER,  INC.. 

Overall  goals:    To-  help  participants  improve  their  economic 
)  condition;  to  prepare  participants  for,  and  get  them  involved 
in  positions  with  lasting  unsubsidized  employment;  to  help 
them  get  off*  and  staff  off  public  assistance. 

Population:    Economically  disadvantaged  .adults  in  the  citj^ 

« 

Means :     Recruitment,  2-4  weeks  of  motivational  training' 

V  ' 
and  pre-employment  awareness,  workshops,  1  to  6  months  of 

on-the-job  training  with  a  private  employer,  supportiv^ 
services. 


IV 


OBJECTIVES/MEASUREMENTS  SHEET 


For  Program, 

Starting: 

Ending: 


OJTUt  Career  Inc. 
October  1,  1980 
September  30,  1981 


Objectives 


< 
i 


1..  757.  will  'be  employed  full-time 

2.  '857.  will  be  either  employed  or  in 

further  training 

3.  1007.  of  those  on  welfare  before 
training  will  now  be  off  welfare 

4.  6  months  earnings  will  *qual 
pre-training -annual  earnings 
or  exceed  it 


Form  or  Document:     /  - 
Ex  i  t  /  Term  ina  t  ion 

5.  Atleast  85%  will  either  complete 
OJT  or  be-  placed  in  unsubsidized 
job  before  completion 

< 

6.  All  placements  in  OJT/Jobs  shall 
be  at  least  at  minimum  wage 

7.  At  least  257.  of  participants 
lacking  GEDs  will  have  completed 
them  \  . 
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Data  Items  on  Form 


Labor  £orce  status 

Are  you  in  school  or 
.training? 

Receiving  public 
assistance? 

Cumulative  'earnings 
since*  training , 
(compared  to  pre- 
program earnings) 


 *  r*&Sr* ' 


>  ^-Results'  ^ 


employed  fuU-time 

7.  in  school  :6r 
^   ~~  ,rraininfc* 


.J! 


V  %  off  welfare 

-  "  '  1  pojstt- earnings  -  % 

of  pre-earnings 


Termination  status 

Wages  per  hour 
-G^D  completed? 


 7.  completing  OJT 

7.  placed  in  unsub- 
sidized jobs 

7.  of  jobs  at  or 
aEove  minimum  wage 

7.  of  original  non- 
5lDs  .completing  GEDs 
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•*s  3.1     EVALUATION  REPORT  FORMAT,  I 

USE:     To  present  -the  results  and  findings  of  your 
PIC's  overall  evaluation, 

A  PIC  Evaluation  Report  should  be  organized  into  the  following 
major  sections: 

1#     .  Introduction  -  describing: 

Program/project (s)  evaluated  / 
Group  that  conducted  the  evaluation 
-  .     Evaluation  approach  •  *  * 

Any  legislation  that  created  the  program/  - 
mandated  evaluation  t, 

2.  Objectives  -  stating: 
<*> 

Objectives  of. the  evaluation' 

Questiops  to.be  answered  through  evaluation 

3.  .  Methodology  -  presenting:  * 

Overall  evaluation  research  design 
Process  for  the  administration  of  the  evaluation  * 
Data  collection. instruments  ^ 
Analytic  techniques 
Data  analysis  results. 

4.  Findings  -  providing:  . 

Answers  to  the  evaluation  questions 
Information  on  the  strengths/weaknesses  of  the 
project  or^prdgram  _ 

5.  *     Recommendations/Conclusions  -  presenting: 

Jlajor  conclusions 
Recommendations ,  if  required 
Limitations  imposed  upon  the  evaluation 

6#       Executive  Summary  -  distilling  the  activities, and 
findings  described  in  the  body  of  the^^valuatiori . 


/ 
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,      V.     PME  RECORDS  r 

This  section  has  been  set  aside  so  that  you  can  house 
documents  which  relate  to  your  PIC's  planning,  management, 
and  evaluation  process  or  system  in  one  place.    We  recommend, 
at  a  minimum,  that  you  keep  copies  of  the  following  items 

here:  _ 

\ 

Written  description  of  .your  PIC's  PME  ^ 
Process  and  Procedures 

PME  Roles  and  Responsibilities  of  Staff-, 
and  Members  » 

Aids  that  you  create  to  implement  the 
^ME  Process 

Your  PIC's  Strategic  Plan 

Appropriate  sections  of  your  PIC's 
Program  Operations  Plan,  Management 
Plan,  and  Evaluation  Plan 

Locating  all  of  the  foregoing  items  here  will  ensure  easy 
access  arid  should  facilitate  their  use  and  interrelationship. 

Most  importantly,  we  encourage  you  to  refer  to  this 
*  section  of  the  Guide  regularly  as  you  implement  your  PIC's 
PME  Process.    Update  it  as  you  have  to  change  goals,  priori- 
ties, plans,  responsibilities,  or ^operating  procedures. 
Also,  record  any  of  your  own  observations  and  comments 
regarding  your  PME  Process  and  keep  them  here.  •  Usin§  the 
Guide  in  this  way  will  ensure  thht  it  serves  as  an  integral 
part  of  your  PME  Process*  and  a  basic  tool  for*  enhancing 
your  PIC's  PME  effectiveness. 
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GLOSSARY 

This  Glossary  consists  of  two  parts.    Part  One,  is 
a  reiteration  of  the  PME  key  concepts  defined  within  the 
text- of  Section  III  of  the  Guide.    Part  Two  defines  some 
of  the  more  common  terms  related  to sthe  employment  and 
training  field. 


-  r 


> 


k 


«>  * 

PAKT  ONE:     pmb  *fy  CONCEPTS  t 
■  ,        (All  concepts  are. located  in  Chapter  JII) 

Key  Concepts 


•  Advisor  Role: 


p&selide  Data 
.  Woxflcsheets: 


^S^a^ca 


cages^Barriers : 

* 

Broker  ftoler1*-  ^ 


Definition  " / 

"   <s  k 

reviewer  and  consultant  to  CETA  4  # 
Prime  b Sponsor .  «       * ' 

forms  designed  to  permit  uniform 
-and  easy  analysis  of  data  gathered 
routinely  through  PIC's  joanagetoent 

reporting  #^stem.  ^ 

1.  *  , 

obstacles  or  reasons  for  .less  4han  •  v 
"successful  performance  in  an  area. 

facilitator*  for § connections  Between 
and. among.  CETA  program  and  others. 


Cause  and  Effect 
Analysis: 


Comparative  Analysis: 


Components : 


study  of  a  number/Of  va4»les  to  — 
.demonstrate  a  causai  re2j«a.onship  ^ 
between  two  -or  more^acqPs . 

'* 

study  of  the  refiatiVe  performance- of  4 
two  or  more  components,  or  projects 
over  time . 


major  specific  activities,  including  ,  ( 
trainings and  other  participant  programs* 
and  employment  generating  -services. 


• 


9 


& 


Component  I^evel 
ftonitoxing-: 

n 

Control  Gyoup: 


-     Control  Limits: 


Alonoiof 

ftutinofti 


r 


^    review  of  major  facet,  (e.g.  clasftrqpm 
'    training,  linkage  projects)  of%PIC 
operations .  * 

group.  whichVtioes  not  participate  in 
aPIC  program  with  characteristics ^ similar, 
to  persons  in  program*  used  at  mechanism 
for  contrasting  impact  of  progr^p  par- 
ticipation. 

.  range  within  which  variance  f rora.  planned, 
level  of  performance  is  tolerable  .y 

i  *' 


t 
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VKey  Concepts 
>age  ?ycj  *  %  ^ 

Corrective  Action  Plan: 


Correct!^  JJgnitoring: 


Corrective,  Monitoring 
System: 


st  Benefit  .Analysis: 


1 


Deficiencies  or  Signi-r 
'  fici^nt  Variations: 


report  which  proposes  method  for 
eliminating  performance  problems. 


follow-up  analysis  to  determine 
why  performance  varied  significantly 
from  planned  level  aid  to  identify 
possible  remedial  actions  to  correct 
situation.  - 


special  procedures  and  methods  to 
provide  feedback  on  the  sta*jwrof  the 
^implementation  of  a  corrective  istion 
plan.  *  * 

study  of  various  measures  of  program^ 
outcome  related  to  co#st  in  order  to 
assess  benefits. 

areas  i'n  which  performance  falls 
significantly  below  that  planned. 


EGS-  Related  Objective": 


>yitter 


Employment.  Generating 
Needs : 


Employment  Generating 
Tactics:  * 


Emplo^er^-Surveys : 


answers  the  question  of  * "how  well" 
in  terms  of^tiipe,  quantitative  measure , 
4  qualitative  value,  and  cost 


a^aJ 
Sift 


sS7 


fectiveness/ef fort  value % 


local  needs  related  to( developing 
activities  or  services  to  create  jobs. 
*oz  to  otherwise  improve  employment 
opportunities  for  CETA  eligible  partici- 
jJants.  1 

*labor  market  analysis;  marketing;  economic 
and  community  development^tnoentive  pro- 
grams (e.g.,  TJTC) ;  and  linkage  creation- 

local  surveys,  which  a  PIC  can  initiate' 
in  order  to  obtain  additional  information 
regarding  the. characteristics  of  the 
labor  market  and  the  needs  of  employers. 


Evaluation  Research  * 
Design: 

* 

4 

Evaluation  Respori- 
^  sibilities: 


detailed  plan  f or , implementing  evaluation, 


roles  and  duties  for  all  individuals 
t^ai  be  involved  in  the  conduct  of  the 
Evaluation'. 


Key  Concepts ' 
Page  Three 

« 

Evaluation  Review 
Process : 


Goal: 


Hypothesis : 


basic  method- and  procedures  to  be 
adhered  -to  in  'assessing  PIC  per  for- 
mance.  , 

broad,  unquantified  statement  of  an- 
outcome  or  result  which  PIC  wants  to 
.  achieve. 

«. 

proposition -that  is  consistent  with  - 
'  known  racts ,  and  therefore  presumed 
as  possibly  true,  but  which  requires 
further  investigation  for  verification. 


Impact  Evaluation: 


Interview  Guide: 


Interim  Indicators: 


Management  Assessment 
Reports: 


Management  by  Exception; 


measurements  to  determine  the  extent 
to  which  the  program  has  brought  about 
participant  Change,  institutional 
change,  or  otherwise  had  art  effect 
upon  the  community  in  which  your  PIC 
operates •  ?  ^ 

format  which  provides  a  logically 
sequenced  aod  comprehensive  list  of  y 
questions  regarding  a  certaih  area 
of  PIC  performance  to  be  asked  in  a 
persona^  interview .<         %  r 

measures  used  to  assess-  the  ohgoing 
performance  of  PIC's  projects;  •components 
and  program. 

quarterly  narrative  reports sprepared 
for  pfurposes  of  Council  decision- 
making describing  PIC  program  per- 
formance  in  major  areas  of  concern  . 

approach  tp  monitoring  which  states 
v  that  corrective  monitoring,  should  be 
restricted  to  exceptional  deviations 
in  performance.  { 


•Management  by  Information 
and  Monitoring  Needs: 


minimum  amount  o^^lita  needed  tp  review 
PIC-  programmatic  effectiveness  and  to 
exercise  control  over  program  • 


Management  Reporting 
Display  Formats: 


cha^t3  for  summarizing  monitoring 
data. to  permit^easy  analysis  and; 
review . '  '  *- 
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Key  Concepts 
Page  Ecflur 


Management  Review    '  ? 
Responsibilities:  ' 


roles,  am  tasks  to  be  performed  by  PIC 
staff  and  mtembers  in  overseeing  and  / 
coordinating  program  operation  plan. 


Master  Plan: 


lon§  term  agreement  between  00L  and 

prime  sponsor.    Describes-iPS  administrative  . 

and  programmatic  arrangements  of  PS. 

Contains  demographic  characteristics 

whf€h  a  PIC  can  review  to  assess  popu-  ' 

lation  groups  and  their  service  needs. 


Monthly  Reports: 


basic  monitoring  d^ta  increments 
;to  be  submitted  by  all  PIC  projects, 


Objective: 


Occupational  Training 
Priorities : , 


Operational  Control^ 
Process: 


^Operational  Issues': 


Operator  Role: 


Organizational  Status 
Issues: 


*a  quantifiable,,  or  in  some  way 
measurable,  expression  of  what  is  to 
occur  specifically  as  a  result  of  a 
programytactic  • 

'  local  growth  occupations  selected  as 
those  for  which  PIC  should  develop 
training %  programs . 

method  for  assessing,  directing,  and 
correcting  PIC  program  operation  on  an 
ongoing  basis. 

Council  by-laws;  meeting  procedures ; 
and  processes  for  determining  rotes  aiid 
responsibilities  of  members  £nd*  staff, 
and  selecting  officers  and  assigning 
their,  duties.  #« 

designer  and  provider  of  direct  services 
either  to  participants*  employees  or 
other  agencies.  • 

fundamental  arrangements-  Ce.g.  position 
within  prime  sponsor ,  \incorpo ration, 
part  of  existing  local  business  organi- 
zation) for  PIC's  operation.  * 


\ 


Outcome  Evaluation: 


Participant  Related, 
1  Objective: 


assessment  of  actual  performance  vs. 
plan  for  fiscal *year# 

•> 

answers  the  questions  of  who,  how  many, 
what  means,  and  primary  results  expected. 


9 
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Key  Concepts 
Page  Five 


Participant  Related 
Tactics: 


classroom  training  --vocajtiorial ,  class- 
room training,  eraployability;  on-the-job 
training;  upgrading,  retraining,  employment 
and  training  ^services ,    and  supportive 
services. 

•Participant  Service  Needs:  training  and  development  assistance 

'  needs  of  the  individuals  in, those 
groups  which  have  been  selected  as 
targets  for  PIC's  programs. 


Performance  Indicators: 


Performance  Standards: 


measurable  factors  (e.g.  Entered  k 
Employment  rate,  number  olK4pbs 
retrained  for  CETA  eligible  parti- 
cipants as  a  jresult  of  Industrial  a  , 
revenue  Ibonds)  indicating  effectiveness 
within  a  given  area  on  which  it  is 
important  tt  set  objectives  and 
standards . 

statements  of  measurable  results  to 
be  acheived  and  used  to  ascertain  ^S* 
sjuccess  or  effectiveness. 


PIC  Admixiistrative 
Structure:  • 


PIC  Delivery  System: 


arrangements  foe  coordinating  and 
managing  PIC's  projects. 

overall  framework  for  providing  PIC 
training  and  development  (participants)/ 
and  employment  generating  projects. 


PIC  Mission  and  Functions: 


basic  purpose, \rbles  and  responsibilities 
to  be  accompli* Red  by  PIC. 


PIC  Operations: 


9 

manner  in  which  a  PIC  functions  both 
internally  and  in  relation  to  significant' 
others  (e.g.  prime  sponsor  and  community)-. 


tpic  Organization: 


PIC  Priorite  Development 
Need  Ajprfas: 


>pfuL 


rocedures  and 
Methods: 


ERLC 


structural  arrangments  and  division  of 
roles  anfKsresponsibilities'  among  PIC 
members  ano^s<aff.< 

key  organizational  and  operational  areas 
in  which  a  PIC  needs  to  improve  its 
performance • 

basic  administrative  processes  and 
guidelines  for  managing  and  directing 
PIC's  efforts. 
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Page  Six 


PIC  Program  Operation 


Plan: 


written  statement  which  documents  PIC's 
performance  standard,  administrative 
structure  and  organization,  operating 
procedure* ,  delivery  system,  work  plans 
for  each  project  to  be  undertaken!  manage- 
ment  plan  and  procedures;  evaluation 
^plan  6  procedures/  6  budget  allocation* 
accounting  procedures. 


PIC  Purposa; 


PIC  Relets) : 


statement' of  ultimate  intent  and 
results  expected  in  the  long  term. 

by  PICT^-  *  •    -  * 

primary-  approach  ( es^^to  performing 
and  discharging  PIC, mission  &  function. 


Slanne<i  vs.  Aplyi^l 
Performance:  N 


Preventive  Monitoring: 


measurements  of  achievement^**  accom- 
plishments-- ag'ainSt  prarset  obje'ctiv«s 
"    and  standards  contained  in  annual 
plan* 

•  regularly  Scheduled  visits  to  projects*/ 
contractors  to  oversae  and  reveiw'.  interim 
products"  and~  work  processes. 


Prime  Sponsor  (PS) 
Annual  Plan: 


Prime  /Sponsor  com- 
.  •prehansive  Employment 
k  Training  Plan  (CETP) 

Projects;  . 


document  which  Prima  Sponsor  prepares 
annually  to  describe 'the  program  for 
providing  activities  and  services  to 
the  elj»gibl4  population*  for  upcoming 
fiscal  year  an4  to  apply  for  Federal  • 
assistance.  #  \ 

application  for  financial  $s*istanq«  co 
pOL.  •  • 


discrate  serivces  and  activities  to 
be  sponsored  through*  PIC  in  order  to 
achieve  ijts  programmatic  objectives. 


Project  Resign: 


objectives  to  be  accomplished,  major 
activities  apd  Services  to.brf  provided, 
action  steps  to  be  taken,  and  basic 
organization  and  management  plan  for 
delivering,  the  project. 


Project  Level  Monitoring: 


review  of  performance  of  individual 
contractor  or  service  deliverer. 


Project  Resource 
Allocation: 


estimation  of  amount  to  be'  expended 
on  each  prp^ect  to  ensure  cost 
reasibility.  and  compliance  with 
original  program  plan . 
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Project  Scope: 


specific  objectives  to  be  accomplished 
and,  services  or  activities  which  are 
to  comprise  each  project* 


Project  Workplan: 


.  Proposal,  Criteria  : 

Relative  Effectiveness 
Evaluation: 


detailed  statements  of  services/activities 
tp  be  undertaken,  planned  inputs  and 
outputs,  work  to  be  performed  by  staff 
organization  charts,  work  schedule  and 
.action "steps,  administrative  procedures, 
and  monitoring  arrangements . 


objective  factors  which  aire  employed  to 
review  responses  to  Request  for  Proposals , 

comparative  analysis  of  projects, 
services  delivered  or  program  com- 
ponents ^undertaken  to  ascertain  the 
relative  merits  of  each. 


Reliability : 


Request  for -Proposal 
.  tRFP) 


extent  to  which  evaluation  would 
produce  the  same  or  consistent  results 
through    repeated  tests  over  tiioe  , 
using  idential  procedures  and 
instruments.  j 

solicitation  used  to  inform  potential 
service  providers  (contractors  and 
subgrantees)  about  the  projects  which 
PIC  intends  to  fund;; 


v     Report  Monitoring: 


Reporting  Requirement? : 


Sample: 


review  of  planned  versus  actual  per- 
formance using  reports  submitted  by 
projects  "and  contractors. 

/ 

frequency  and  nature  of  formal  repprts 
to  be  submitted  to  permit  effective 
program  management  and  monitoring. 

a  part  of  a  statistical  population  whose 
characteristics^:  opinions  *re  studied 
to  gain  information  about  the  whole. 


Service  Deliverers: 


4  O 
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any  organization  or  firm  the  PIC  uses 

t-o  provide  services  or  perform  activities. 
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Service  Providers: 


government  agencies,  community  based 
organizations , businesses ,  and  other  con- 
tractors who  furnish  or  might  provide 
assistance  to  PIC. 


Staffing  Issues: 


arrangements  for,  level,  nature  and 
division  of  labor  among  those  individuals 
to  provide -Staff  services'  for  PIC; 


Strategic  Decision-Making 
Process:  >> 


Strategic  Plan ; 


v 


method  which'*  PIC  chooses  to  employ  to 
select  its  goals  and  to  develop  its 
strategic  plan. 

summary  statement  which  (1)  details  all 
the  goals  PIC  wants  to  accomplish  in  the 
upcoming  year  expressing  their  relative 
importance?  and  (2)  presents  longer  term 
t6rm  go&'ls  and  establishes  general  time 
parameters  for  their  accomplishments. 


Structural  Issues: 


size  and  composition,  membership  selection 
processes,  and  committee  types  fo*  PIC. 


Subparts: 


Survey  Questionnaire: 


major  segments  of  the  Prime  Sponsor's 
Annual  Plan. 

instrument  designed  to  collect  data, 
opinions,  or  attitudes  regarding  aspects 
of'  a  PIC'S  program. 


Survey  Research: 


Tactics 


.reliable  methxJi*  employing  some  form  of" 
questionnaireVafir  generating  data  per- 
taining to  respoimant  experience, 
attitudes,  arid,  opinions  related,  to  an 
area  of k impact  of 'PIC  performance. 
I 

-general  means  or  broadly  formed,  approach 
for  achieving  a  goil  or  an  end. 


Target  Groups: 


those  significant  segments  (e.g.  youth, 
blacks,  welfare  recipients,  displaced 
homemakers)  of  the  local  population 
who  are  selected  as  the  foci  for  PIC's 
programming  efforts. 
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Terminal  Indicators: 


Trend  Analysis: 


Validity: 


* »  . 
measures  used  to  assess  the  end  results 
or  outcomes  of  a  project,  component, 
or , program . 

study  of  performance  over  time  to  see  if 
it  is  increasing,  decreasing,  or  staying 
the  same • 

extent  to  which  ^alua?ionJC^Ua^LUre 
measures  what  it  is  intended  to  measure 
taking  all  important  factors  into 
account.-  ' 


Warning  Light: 


indicator  that  project  has  exceeded  control 
limit  -for  a  given  area  of  performance. 
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PART  TWO:    Employment  and-Training  Terms 


A-95  Review  Process: 


AA: 
AAP: 

« 

ABE:  • 

Adult  Bafcic  Education  (ABE) : 
'  Adult  Work  Experience  (AWE) : 


Affirmative  Action  (AA) 


>  9 


Affirmative  Action  Plan 
'  (AAP)  : 

Applicant  pool: 

Appropriation: 


Area' of  Substantial 
Unemployment: 

'  Si 


/ft  National 
AVAfcnat  of 
£r\x  Business 
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federally  mandated  process 
where  applications  for  federal 
funds  are  reviewed  by  state- 
wide clearinghouses  so  local 
governments  can  assess  the  . 
interrelationships  and  foster 
cooperation.    The  name  A-95 
derives  from  its  source  as  OMB  -       ^  ^ 
Circular  A-95 

.Affirmative  Action 

Affirmative  Action  Plan 

Adult  Basic  Education 

GETA  classroom  training  program^ 
which  teaches  basic  skills  to 
adults 

CETA  program  activity  for  adults 
designed  to  provide  a  familiari- 
zation period"  for  those  new  to 
or  re-entering  the  labor  market. 
Activities  included  part-time  work 
and  exposure  to  work  habits  and  % .  - 
career  possibilities 

— Ehe"~ active  effort  to  eliminate,, 
barriers  to  employment  opportu-  . 
nities  for  minority  and  female 
workers;  required  by  law  in  all 
federally  funded  projects  t 

formal  statement  of  an  organization's 
approach  to  reach  its  Affirmative. 
•  AGtion  goals  ,  .  . 

-   pool,. or  group,  of. potential 
applicants  , 

specific -amount  of  money  disignated  ' 
by  Congress  for  a  specific  program. 
Allocations  cannot  be  made  without 
a, Congressional  appropriation 

a  specif  ideographical  area  with 
an  unemployment  rate  in  excess 
of  6.5%.    This  classification 
.  is  sometimes  us*ed  to  determine 
-  area  eligibility  for  pufilic  serr 
vice,  employment  programs  > 
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Page  Two 


^Assessment : 


Autonomy : 


AWE:  - 

Balance  of  State (BOS) : 


Balanc^l  Placement  Formula; 
(BPF) 


BAT: 


BIS: 

Block  Grant:  - 


r 


BLS :      .  * 
BOS: 

» 

Budget  Information  Summary: 


Bureau  of  Labor  Statistics: 
(BIS) 


CBO: 
CETA: 


/onq^l  estimatidn  of  CETA  ' 
client's  skills  and  training 
needs 

the  quality  of  state  of  self- 
governance,  self  control/ self 
.containment ,  independence 
from  the  whole  ' 

*  adult  work  experience 

the  area  outside  the  boundaries 
of  the  CETA¥prime  sponsdrs,  in 
a  given  state  and  generally 
composed  of  smaller,  rural  counties 
under  the  administration  of  the 
prime 'sponsor  which  allocates 
CETA  monies  to r projects  in  those 
areas 

formula  used  to  provide  funds 
to  the  United  States  Employment 
Service 

Bureau  ©f  Apprerfticeship  and 
Training 

Budget  Information  Summary 

.  a  jjrant  of  federal  funds  to  a 
local  government  to  be  used, 
within  limits,  as  >the  local 

v  go\rernment  decides 

Bureau  of  Labor  Statistics 

Balance  of  State 

a  CllTA  report  which  shares  the 
prime  sponsor's  or  PIC ' s*  planned 
expenditures  by  program  activitv 
and  cost  category;     the  BIS  is; 
part  of  the  annual  plan  m 
* 

a  division  of  the  U.S.  Department 

•  of  Labor  which  publishes  socio- 
economic data  including  manpower  * 
and  cost  of  living  statistics 

community  based  organization 

Comprehensive  Employment  and 
Training  Act 
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CETA  Planning  Council: 


CETA  Planning.  Cycle: 


CETA  Prime  Sponsor  (PS) : 


Chief  Elected  Official : 
(CEO) 


Classroom  Training: 
(CT  or  CRT)*: 


.crten^: 


Client  Characteristics: 


1 


appointed  by  the  Chief  Elected 
^Official  to  advise  in  overall 
pl-anniag.of  the  employment  and  % 
training  programs  under  all 

♦  titles  of  CETA  . 

the  time  period  in  whi^ch  planfe 
.for  CETA  programs^  proposed  fbr  . 
the  next  fiscal  yea%vJ£*?tober  1  - 
September  30)  are  developed  and 
submitted  to  the  Secretary,. 
^Department  of  Labor  for  approval. 

•  •Key  dates  are:    May  15  -  June  1, 

'planning  estimates  and  revisions 
to  regulations  issued;  July  1/ 
preliminaryvplan  submitted  to 
pepartment  of  Labor,  Regional  • 
'  Office  and  immunity  review  be- 
gins August  14,  review  and  comment 
period  ends  and  plan  is  revised; 
September  1,  final  plan  submitted 

aTState,  county,  local  government 
or  consortium  unit  acting  aV 
administrator  of  the  Comprehensive, 
Employment  and  Training  Act  funded  . 
programs  in  th6* designated  area 

^lighest^  elected  official  of  a 
.  given  governmental  entity.  Usually 
the  mayor  of  a  municipality,  the 
governor  of.  a  state,  or  the  co\inty 
executive  of  a  county.    The  Chief 
Elected  Official  of  each  CETA 
jurisdicatipn  is  legally  responsibl 
for  the  program  in  that  arGa , 

av  CETA  program  activity"  in  which  . 
training  is  conducted  ifl  an  insti- 
tutional setting  to  pro^de  the 
technical  skills  and  information 
required  to  pet form  a  specific  job 
or  to  upgrade  basic  Skills 

a  person  enrolled  ;in  the  CETA 
program 

• 

demographic  characteristics  of  a  . • 
client.    As  part  of  the  grant 
application  to  the  Department  of 
Labor,  each  prime  sponsor  selects 
a  series  of  characteristics  that 
define  and  describe  the  profile 
of  fliients  the  prime  sponsor  will 
Ve 
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Client  Community: 


Community  Based  Organization; 
(CBO) 


Comprehensive  Employment 
and  Training  Act   (CETA) : 


Comprehensive  Employment  and 
Training  Plan: 


Contractor: 


Corrective  Action: 


Counseling: 


the  groite  of  people  to  be 
served  by  a  prpgram  activity:  •* 
those  in  need  of  services 

a  private  non-prof it ^organization 
which  is  representative  of  the 
community,  of  a^  significant  element 
of  the  community;  may  provide  employ- 
ment St  training  services  or  activities 

an  act  by  Congress  "to  provide 
job  training,  and  employment  * 
opportunities  for  economically 
disadvantaged ,  unemployed  or  under- 
employed persons  which  wi^ll  result 
in  an  increase  in  their  earned 
income,  and  to  assure  that  training 
and  other  services  lead  to  maximum 
employment  Opportunities  and 
enhance  seif  sufficiency"  • 

*"  •  w 

a  master  and  annual  'plan  describing  ■ 
the  administration  and  programs 
under  CETA  which  is  developed,  by 
tbe  prime  sponsor  with  the  parti- 
cipation of  the  CETA' Planning  *'  * 
Council  and  submitted  to  the  Secre- 
tary/Department of  Labor  for  .   •  % 
approval 

for  CETA  purposes,  any* person , 
corporation  or  organization  which 
enters  into*  a  contract  with  the  , 
Department  of  Labor  /  a  pzime^  sponsor 
or  a  PIC  .to.  provide,  sprviceg 

action  necessary  to  correct  iden* 
tified  deficiencies <  « 

*       *  *  * 
for  CETA  purposes,  career  and  personal 

consultation^or  CETA 'clients  with 

professional  CETA  counselors 


CRT:  '  classroom  training 

CT:  classroom  training  , 

Current  Population  Survey:  a  monthly  household  survey  of- 

a  sample   (sixty  thousand)  of  the 
civilian  non-institutional  population 
of  the  United  States;  conducted 
by  the  Census  Bureau.    The  survey 
•      -  provides  monthly  statistics  on 

employment ,  unemployment ,  and  ' 
#'  .  related  subjects  analyzed  by  BLS' 
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# 


Data  BAse: 

>  _ 

Decentralization : 


•Delivery  System: 


*  Department  of  Housing  and 
Urban  Development  (HUD) 


Department  g£  Labor: 

(DOL)  ^      *  3 


Dicticrtiary  oi  Occupational 
Titles  '(DO?/bOT  Code) : 


direct  Placement: 


Disadvantaged : 


ttfie  pool  of  information  collected 
and  retained  by  a  computerized 
or  manual  information  system 

the  delegation  of  greater  authority 
and  responsibility  for  the 
operation  and  planning^pf  pro- 
gramf  to  units  of  state 'and .local 
government •    CEJA  is  a  decentralized 
system  m  \ 

th$  coordinated  system  of  providing 
services  to^a  client  comidunity, 
including  planning,  administration, 
service  provision,  etc. 

federal  agency  responsible  for 
addressing  the  housing  and  urban 
problems;  administers  -the  Community  * 
Development  Block  Grant  and  other, 
programs 

federal  agency  responsible  for 
, administering  the  C£TA  program 
through  its  Employment  and  Training    •  % 
Administration  (ETA}  .    ETA  receives, 
the  Congressional  appropriations , « 
allocates  funds  to. prime  sponsors, 
across  the  nation,  issues  regulations 
ba?e&  on  the  Act  and  mbnitors  the  , ^ 

prime  sponsors  for  compliance. 
DOL  also  administers  a  variety 
of  other  , programs ,  including  the 
U.S.  Employment  Service 

the  classification  of  occupations, 
through  the  use  of  a  six-digit 
numerical  code,  wit;h  specif ic 
■   job  descriptions  " 

the  placement  of  a  CETA  client 

by  a  CETA  prime* sponsor ,  PIC  or  ^  v 

contractor  directly  into  unsub- 
m  sidized  employment  without  having 
M provided  the  client  with  a  program 

activity 

a -person  or  persons  who    possess  age, 
racial,  educational,  physical,  or 
mental  characterises  which  set  tihem 
apart  from  the  c6mpetitive  market. 
These  characteristics  are  usually 
combined  with  income  guid^ines  ana 
current  labor  market  status  indicators 
in  definitions  designed  for  administrate 
and  planning  purposes 


•er|c 


Hi 


"Employments*  Training  * Terms 
Page  Six 


' Discouraged  Workers: 


« 

» 


r 


DOL:  „  ' 

.DOT/DOT » Code:       '  . 

Drojsout : 

Economic  Development: 

% 

« 


Economic  Development  Act: 
/       \      (EDA)  " 


^m*,        Economically  Disadvantaged: 


rERIC 


EDA: 
EEO: 

Eligibility; 
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persons  without  work  who  to^fce 
no  oyert  attempt  to  (find  a  • 
*job  because  tljey  feel  no  wo'rk  • 
isr  available  to  them.  Often 
ref err^d^to  as  the  "hidden 
^employed*,  "they  are  .not  in- 
cluded in  the  unemployment 
estimates.    The  withdrawal* 
of  these  persons" from  th6  labor 
market  during  periods  of  high 
unemployment  results  in  an  under- 
estimation of  the  -severity  of 
^j^iplo^ment^  *  • 


Department  of  Labor 


Dictionary  ^>f  Occupational" 
Titles  ** 

person  who  leaves  school  or  a 
training  program^before  completing" 
a  specified  program  „  •      '  ' 

*  * .  <      «  • 

the  development  of  an  area's 
economic  health,  throif^h  such 
activities  as;  assessing  the*  ^ 
economic  climate  and" using 
government  programs  to  encourage  , 
business-  expansion  and  footer 
government  cooperation  with 
private  industry* 

-  federal  la\^setting  up  the    ]  , 
Economic  Devel&pment  Administration 
in  the  Department  of  Coit&nerce* 
and  establishing 'programs  to 
encourage1  ecbnolflic  development 
by  funding  seed  projects,  public 
works,  etc. 

t  •  t 

an  individual  who  is  a  "member  of    m  %  [ 
a  -family  i  (1)>*  which  receives  cash 
welfare  payments;"  or  (2)  whose 
'  annual  inc6ine  in  relation  to  family 
£ize  do$5  not  exceed  thq  poverty 
lev^l  guidelines  issued  by  the  • 
federal  Office  of  Management  and  *  . 
Budget  ^  1 

.  Economic  Development  Act  or  Economic, 
,  Development  Administration 

equal  employment  opportunity 

criteria  used' to  enroll  people 
into  a  program  such  as  C?T/ 
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feliqible:* 

-  I 

fimployability: 


Emplfcy^bility  Development: 


Employment  and  Training  ( 
Administration  (ETA)  >: 


qualified  for  (meets  the  criteria 
k       for)  a    program  such  as  CETA 

qualities  which  enhance  an 
individual's  ability  to  be  employed, 
#      such  as  skills/  educatiotT and 
experience 

« 

the  effort  to  provide  training 
,    arid,  other  services  to  people 
in  brder  to  enhance  their 
employability  or  thejLr  marketable 
skills  and  apititudes  ,< 

a  division  of  the  U-  S.  Depart- 
ment of  Labor,  that  is  tjie 
administering  agency vf6r  CETA 
programs  on  a  nationwide  basis 
operating  through  10  regional 
*  offices 


Employment  Security  Automated 
Reporting  System  (ESARS): 


EJigLish  a*  a  Second  Language : 
(ESfc) 


Equal  Employment  Opportunity: 


ESARS 


ESL  ' 
ETA 


f External  labor  Market^ 

FRir# 


series  of  reports  produced  by 
the  State  Employment  Security 
Agencies  which  give  labor  market 
information  and  breakdown  of 
characteristics  of  people  being 
seryed  by  the  Ergployment  Service 

a  CETA  program  activity  which 
provides  classroom  training 
to  teach  CETA^clients  with  »a 
native  language  other  than  English 
an,  ability  -tto  use  the  English 
language  \. 

the ^provision  of  equal  opportunities 
for  employment  without  regard  to 
race,  creed,  color,  handicap, 
/national-  origin.,  sex,  age/ political 
affiliation  or  beliefs  t 

fcmploymeilt  Security  Automated 
Reporting  System 

'  English  as  a  second  language  * 

Employment  and  Training 
Administration  ,  / 

' •         that  market  lying  outside  >the 
643  particular  firm  or  craft.     It     r  ^ 

is  relatively  free  of  administrative 
rul^s  governing  the  pricing  and 
allocation  of  labor/por,ts/of  entry 
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Federal  Representative 
Fed  Rep  (FR) : 


Fed  Rep: 

Federal  Assistance  Program: 


Federal  Managentent  Circular: 
.    (FMC)  ' 

1  Federal  Register:-  ' 
(FR) 


Federal  Regulations: 


Field  Memorandum  (FM) : * 


individual  from  a  regional 
U.  S.  Department  of  Labor  office  # 
assigned  responsibility  for 
employment*  and  training  programs 
Within  'a  defined  geographic  area 

Federal  Representative 

a. term  used  to  refer  tQ  the 
variety  of- federal  programs 
available  to  state  and  local 
governemnts  including  counties, 
cities,  metropolitan  an£  regional 
governments;  schools,  colleges, 
and  universities;  health 
institutions,  non-profit  and  fox 
profit  organizations;  aijd  to 
individuals  'and  familie^.  Current 
federal  assistance  'progtrams  'are 
listed  in  the  annual  Catalogue 
of  Federal  Domestic  Assistance. 
Federal  assistance  programs,  provide  , 
assistance  through  grant  or  con- 
tractual arrangements  aiid  include 
technical  assistance  programs  or 
programs  providing  assistance  in 
the'  forms  of  loans.    CfeTAis  one 
of  these  programs 

a  directive  issued  by  the  federal 
Generals-Services  Administration 

publication  which  carries  the  text 
of  federal  regulations  as  they  are 
issued  by  federal  agencies:  Often 
draft  regulations  appear;  for  com-* 
ment  and  review,  thqn  final  or 
"implementing  regulations "  Appear 

• 

operating'  procedures  for  federal*; 
programs  based  on  federal ^legislation 
issued  by  the  appropriate  federal 
agency.    Agencies  receiving  federal 
grants  are  required  to  comply  wit^i 
^bhe  arpjiropriafce  regulations 

national  directive  issued  by 
•0 .5 .  "Department  of  Labor 
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Fiscal  Year  (FY) : 


FM: 

FMC: 
Follow 


FY: 


General  Educational  Develop- 
ment (GED)  : 

I*  Hands-on-Ekperience:  . 
Indirect  Placement: 


Intake \ 


Internal  Labor  Market: 


.    job  Development: 


Job  Placement:  ; 

-  \  •  < 


year  for  financial  reporting 
purposes  (not  the  calendar  year) • 
The  federal  government  and  all 
'  CETA  funded  programs  operate 
on  October  1  through  September  30 
fiscal  year 

Field  Memorandum 

Federal  Management  Circular 

contacts  made  by  representative 
of  a  service  delivery  agency  with 
former  clients  to  determine 
additional  services  needed  by 
the  client  or  to  collect-  infor- 
*  mation  on  the  client's  current 
.  status  for  evaluative  purposes 

Fiscal  Year 

qualifying  test  for  high  school 
diploma  equivalent  « 

work  exjJ^rience  where  one  uses 
actual  equipment  normally  used 
on  the  job;  also  may  mean  clo^e 
simulation  of  tasks  as  a  training 
-  method 

'  the  placement  of  a  CETA  "client  , 
by  a  CETA  prime  sponsor,  PlC 
or  contractor, into  unsubsidized 
employment  after  the  client  has 
been  provided  a  program  activity 
or  supportive  service 


t 


the  process  of  bringing  clients 
into  the  CETA  system 

that  labor  market  within  a  parti- 
cular industry,  firm- or  plant 
in  which  the  pricing  and  allocation 
of  labor  are  governed  by  admini- 
strative rules  .and  procedures 

the  process  of  developing  or  finding 
jobs  for  CETA  clients 

the  process  of  placing  CETA  clients 
in  jobs 


"ERIC*    •  * 
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Job  Related  Education  (JRE) 


JRE: 


refers  to  the  education  necessary 
to  enable  a  worker  to  assimilate 
OJT.    JRE  bay  include  basic 
remedied  education  in  mathematics 
and  language  skills  needed  to 
perform  job  duties •  Technical 
information  may  be  included/ 

job  related  education 


Labor  Market: 


Labor  Market  Area  (LMA) 


LMA: 
LMI: 

Low  Income : 


for  purposes  of  state  or  local 
manpower  planning,  the  geographicsil 
area  within  which,  most  workers  are 
secured;  the  geographical  are^y 
over  which  a  worker  can  roam  in 
search  of  a  lob,  within  reasonable 
commuting  dia^ajice  of  his  place 
or  residence 

definable  afea  in  which  there  is 
considerable  ^economic  integration 
and  within  whach  workers  cam  and 
do  readily  ^effai^e  jobs  without 
changing  their\ftia£es  Qf  residence 
usually  a  central  city  or  cities 
and  the^  surrounding  commuting  area 
**. 

labor  market  area 

labor  market  information 

income  in  relation  to  family  size, 
determined  by  BLS  for  farm  and 
-rOJ*  non-farm  ^families  I 


Lower  Living  Standard  Income       income  level  ill  relation  to  family 


Level : 


Maintenance  of  .Effort: 


size,  determined  by  the  BLS  fo'r 
farm  and  non-farm  families 

a  requirement  undfer  CETA'that 
CETA  fund  a.  nqt  be  used  to  displac 
currently  employed  workers  or  to 
replace  existing  services  funded 
from  non-CETA  sources  • 


Manpower : 


activities  designed  to  place 
people  in  jobs  or  to  improve  their 
employ ability  and  the  job  opportunities 
available  to  them1.    The  terms 
v  "Employment  and  Training"  have 
i  generally  Replaced  the  term  wM«i- 
)  power" 
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NAB  ,        .  . 

National  Alliance  of  Business 


OEDP 

Overall  Economic  Development 
Plan 

k 

0JT 

On- the -Job  Training 

( 


Participant 

P^C  (PSIP)  Annual. .Plan 


PPS' 


National  Alliance  of  Business 

a  business-led,  non-profit 
corporation  working  in  partner- 
ship with  business,  labor, 
gpvernment,  education  ar$ 
community  groups  to  reduce 
4in  employment  problems  of  eco- 
nomically disadvantaged  youtfrs 
and  adults  ■  • 

Overall  Economic  Development  Plan 

< 

plan  required  by  the  federal 

Economic  Development  Administration 

in  order  for  municipalities  to 

receive  funding- for  public  works 

projects  and  other  grants  from  EDA  • 

if 

on-the-job  training 

CETA  program  activity  in*  which  a 
participant  is  actively  employed,  m 
at  a  normal  wage,  by  a  subsidized 
contractor  and  is  trained  at  the 
work  site  toa perform  certain  jobs. 
The  employer  contracts  with  CETA 
to  train  the  participai\t  and  CETA. 
reimburses  the  employer  for  the 
training  costs 

an  individual  who  is  enrolled  into 
CETA  and  is  receiving  services  or 
taking  part  in  program  activities* 

the  Title  Vir  subpart  to  the  CETA  , 
prime  sponsor's  Comprehensive 
Employment  and  Training  Plan  which 
must  ,have  -concurrence  of  both  the 
PIC  chairperson  .and  Chief  Elected 
Official  of  the'  CETA  prime  sponsor 
in  order  to  be  approved  by  the  _  * 
Department  of  Labor      (20  CFR  679.5) 

Program  Planning  Summary 

*     .  /  . 
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Prograi#Planning  Summary 


Program  Status  Summary 


PS 

PSE 

PSS 

Public/Private  Partnership 


C  •      •  • 

Public, Service  Employment 


RA 


Regional  Administrator 


Regs 

V 

Regulations 

Request  for  Proposal 

RFP 
SESA 
SIC    ■  r 


a  CETA , report  which  shows  the 
prime  sponsor's  planned  enroll- 
ments by  program  activities?  the 
PPS  is  part  df'the  annual  plan 

v 

a  CETA  report  which  shows  the 
prime  sponsor 's  actual  enroll- 
ments by  program  activities, 
compared  to  its  planned  enroll- 
ments (as  shown  on  the  Program 
Planning  Summary) 

prime  sponsor 

public  service  employment 

program  status  summary 

a  formal  or  informal  agreement  ^ 
between  two  sectors  to  define 
and  resolve  community  problems 
for  their  mutual  benefit 

CETA  program  activity  which  f 
subsidized  employment  for  CETft^s 
participants  to  work  with  public 
employers  {governmentl  and  private 
non-profit  employers  By  providing 
public  services      (Titles  II-D 
and  IV)  '  '  • 

Regional  Administrator 

the  chief  regional  administrative 
officer  of * th£  U.S.  department  of 
Labor.    There  is  on  Regional 
Administrator  per  federal  region 

see  Federal  Regulations 

see  federal  Regulations 

4 

formal  process  of  soliciting 
proposals  for  a  project,  program 

*   or  contract  '  „ 

t 

request  for  proposal 

State  Employment  Security  Agency 

Standard  Industrial  Classification 
Code  J 


ERIC 


MS 
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Significant  Segments 


SMSA  -  ^ 

Standard  Industrial 

Classification  Code  (SIC) 


Standard  Metropolitan 
Statistical  Area  (SMSA) 


State  Employment  Security 
Agency  (SESA) 


Supportive  Services 


those  groups  of  people  characterized 
by  racial,  ethnic',  sex,  age, 
occupational  or  veteran  status, 
which  causes  them  to  experience 
unusual  difficulty  in  obtaining 
employment  and-  therefore  are  most 
in  need  of  series.    DOL  or  the  ^ 
prime  sponsor  may  designate  a 
significant  segment  as  a  target 
group  to  receive  priority 


) 


Standard  Metropolitan  Statistical  Area 

a  system  for  the  classification  and 
description  of  employers  by  type^ of 
industrial  activity  in.  which  th^y 
are  engaged.    The  SIC  is  published 
by  the  federal  Office  of  Management 
and  Budget  and  regularly  updated 

a  widely  used  concept,  developed  . 
jointly 'by  the  federal  Departments 
of  Labor,  Commerce  and  Health  and 
Human  Services  for  defining  labor 
market  areas  for  the  purpose  of 
compiling  statistics.    SMSAs  are 
definable  areas  in  which  there  is 
considerable  economic  integration  ^ 
and  within  which  workers  can  and  ao,,, 
readily  change  jobs  without  changing 
their  places  of  residence.    The  % 
Bureau  of  Labor  Statistics  (BLS) * 

*    -  establishes  lower  living  standard 

/    income  levels  by  SMSAs 

a  state  administered  agency  funded 
-   through  the  Department  of  Labor 
'which  offers  jof>  service,  placement 
and  counseling^ provides  labor 
market  information ,  jmd.  administers 
Unemployment  Insurance;  coordinates 
other  programs  such  as  Job  Corp, 
.   Work  Incentive  Training  (WIN)  ana 
apprenticeship  training  programs 

services  provided  to  participants 
which  are  not  manpower-related  but 
which  are  necessary  to  enable 
'     individuals  to  participate,  including 
-health  care  and  medical  services, 
child  care,  transportation,  resi- 
dential assistance,  assistance  in 
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TA 

Targeted  Job£  Demonstration 
Program  (TJDP)     *  ■ 


/ 


Targeted  Jobs  Tax  Credit 
- (TJTC) 


Target  Groups 


Technical  Assistance 


ERLC- 


securing  bonds,  legal  services 
and  family  planning  services 
(provided^#uch  services  are  made 
available  to*, a  participant  only 
on  a  voluntary  basis 'and  ate  not 
a  prerequisite  for  participation  * 
in  or  receipt  of  any  services  or 
.benefit  from  the  program) 

tedhfiical  assistance 

grants  from  DOL  made  on  a  selective 
baslfe  to  units  of  local  government 
to  promote  employment  opportunities 
for  the  economically  disadvantaged 
through  funding  of  economic  and 
community,  development  projects 

an  elective  tax  credit  that  applies*4 
to  wage  costs  of  eligible  employees 
between  January  1,  1979  and  * 
December  31,  1981,  and  is  equal  to 
50%  of  the  first  year  wages  up  to 
$3,000  and  25%  of  second  year  wages 
up  to  $1,500.    Youth,  ex -of fenders 
and  the  economically  disadvantaged 
are  eligible,  in  addition  to  other 
groups  as  determined  by  the  State 
Employment  Service 

population  groups  identified  as 
having  special  barriers  to  employ- 
ment and  therefore  given  priority-, 
to  receive  services.*  See  Significant 
Segments 

a  term  used  to  refer  to  the  programs , 

activities,  and  services  provided  by 

the  federal  gpvernment,  a  public 

interest  group,  or  another  third 

party  to  strengthen  the  capacity  of 

program  operators  (PICs,  prime 

Sponsors ,  'contractors ,  etc . )  to 

improve  their  performance.    The  * 

delivery~~6f  technical  assistance 

requires  serving  one  or  more  of  * 

three  functions:     (1)  transferring 

information;   (2)  developing  skills; 

and  (3)  transferring  products.    Ther  ^ 

tools  of  TA  include  counseling r  ~ 

training f  giving  expert  information, 

providing  equipn*ent,  goods  or  services^B 

including  advisory.  Technical 

assistance  mayTSe  provided  onsite 

or  in  a  training  session 
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TJDP  ' 
TJTC 

UI  £ 

Underemployed 


Unemployed 


Unemployment  Insurance 
(UI) 


Unemployment  Rate 


Unsubsidized  Employment 


Vocational  Education 


WE 

Work  Experience 


Work  Force 


Targeted  Jobs  Demonstration  Program 
Targeted  Jobs  Tax  Credit 

c 

unemployment  insurance  t 

persons  working  full  or  part-time 
with  inadequate  .incomfc*.  This 
includes  thoset  working  |>art-time 
because  full-time  work  is  unavail- 
able, those  working  full  time  with 
earnings  that  do  not  raise  family  t 
income  above  the  poverty  or  near- > 
poverty  level,  and  those  whose 
abilities  would  qualify  them  for 
•higher  jobs  when  such  are  available 

a  person  who  is  without  a  job  and, 
wants,  and  is  available  for  fcbrk 

a  program  of  cash  benefits  available 
to  insured  unemployed  workers  through 
the  State  pmployment  Security  Agencies 

i   . 

the  number  of  people  unemployed, 
expressed  as  a  percentage  of  the 
civilian  fabor  force 

jobs  Jhich  are  not  subsidized  by 
government  manpower  funding 

training  is  specific  skilled  •' 
fields,  usually  through  an 
accredited  vocational  educational 
institution,  sucfras  vocational 
high,  school 

work  experience 

a  CETA  program  activity  designed  - 
to  provide  a  familiarization  period  t 
for  those  new  to  or  re-entering  the 
labor  market.    Activities,  include 
part-time  work,  and  exposure  to  , 
work  habits  and  career  possibilities 

total  number  of  persons  employed, 
,  based  on  establishment  data  rather 

than  census  data .    Because  these 

statistics  are  derived  from  surveys 
•  of  employment  establishments,  they 

differ  from  labor  force  statistics 
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Youth  Work  Experience 


that  are\ based  on  household  data. 
The  difference  between  work  fo^e 
and  labor \force  statistics  is 
particularly  significant  when 
data  are  being  compared  for  places 
where  workers  commute  between  areas 

'a  CETA  prograp  activity  .for 
youth  designed  tp  provide  a 
familiarization  period  for 
those  new  to  the  labor  market. 
Activities  include  ^art-time 
work  and  exposure  to  work  habits  f 

-and  career  possibilities 


YWE 


youth  work  experience 


> 
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This  Appendix  presents  a  selected  bibliography 
of  references  which  you  might  refer  tovielp 
you  in  implementing  your  PIC's  PME^rocess, 
It  lists  all  of  the  primary  references  cited 
in  Sectional  of  the  Gjyjje.' 
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